As some of you might have noticed from the lack of new articles in the last months, I've been into a bit of a rough patch. Nothing dramatic, but a series of other work and personal commitments lined up and managed to keep me away from Particular Reality.

Unfortunately, by its nature of no compromises passion project (you can read more about this in my article "How to pick a game project"), it can happen that I need to put it on hold.

But what was supposed to be a short break of a couple weeks became a six months limbo.

Here's the graphic summary of year one and year two, in a format similar to the one I used one year ago.

Basically, in a whole year I managed to only do 11 weeks of part time work on the project (probably around 250 hours total). It really wasn't a good year, I guess. Fingers crossed for the upcoming one.

But what went wrong? I'm going to try and discuss the core problem...

The high cost of context-switching

I'm not talking about processes executed by the CPU, but it's actually quite the same: being forced to stop working on a project for a while makes you forget many details (the "context"), so going back to it can feel a daunting task, because you must first "rediscover" them.

Keeping the code well organized can definitely make the process easier and faster, and the DevLog itself is helpful too: I can read the last couple of posts, and remember what was I working on.

But when you are experimenting with features, and doing "exploratory" programming, you can't expect to always have perfectly organized code. So, if you are forced to stop working at the wrong time, resuming can be a bit more challenging than it should be.

I went into a downward spiral where I initially didn't go back to the project because I didn't have sufficient time to "get back into it", but that implied staying away from it for longer time. Which, in turn, made that process of "getting back to it" progressively need more time and effort. Because of other work deadlines, I could never manage to get back on track. I tried a couple of times, but things happened and I had to stop again. Basically, six months went by in the blink of an eye.

So what's the plan?

After six months of not opening the project at all, I admit that I don't feel like resuming work right now, even if I have a little more free time in the upcoming three weeks. My current priority is completing and archiving some other tasks I've been neglecting for too long, so that finally, after that, I will hopefully be able to focus on Particular Reality without other interruptions for a long time. The idea of resuming work for a bit, and then having to take another break to take care of something I have on my backlog, is unsustainable.

So, even if I prefer to not set a specific "resume" date yet, I still wanted to publish this quick "birthday" post to explain what's (not) going on.

The project is not dead and I'm still thinking about it and occasionally writing down some notes that will come useful.

Zero doubts about wanting to keep working on it and publish the final result.

I just need to clear the road ahead first.

Until next time!

I'm going to proceed like I did in week 33 for the limbo zone, but this time dealing with the menu zone.

Let's see what I designed in week 25, and in particular on 2024-08-27:

...after entering from the limbo, [the player] will be drawn to the zone where they can open a portal to the adjacent platform. When in position, the floating particles will morph to the gesture needed to open the portal, prompting the player to do that same gesture.

When the "skeleton avatar" hands match the "particles hands", they will activate the portal to the adjacent platform.

...

the player should take a step forward while sticking to the same gesture. We should teach this by having the "particles hand" go forward into the portal, pushing the player to follow in the same way.

Then, this sequence needs to be repeated twice, one to go to the new game platform and another to actually enter the gameplay session crossing a bridge portal.

I also did this sketch showing the three steps:

Let's draft the code for this zone.

Basically, I need to set the same type of hint, but on three different spots.

At least for now, it's totally fine to hardcode these spots.

In the menu zone, I had labelled only the start, new_game and continue_game platforms.

I added a center label to the... well, central platform, so that I can easily look it up from the zone logic.

Skipping the lookup of the zone and platforms, this should basically be it:

            RTLocation playerLoc = m_rPlayerStateManager.getCurrLocation();
            Hex.EDir? portalOpeningSlice = null;
            if (playerLoc.m_platformId == startPlatform.m_id) {
                portalOpeningSlice = Hex.EDir.S;                
            } else if (playerLoc.m_platformId == centralPlatform.m_id) {
                portalOpeningSlice = Hex.EDir.SE;
            } else if (playerLoc.m_platformId == newGamePlatform.m_id) {
                portalOpeningSlice = Hex.EDir.S;
            }
            if (portalOpeningSlice.HasValue) {
                rSmartParticlesSM.setPortalHint(
                    new RTSpot(playerLoc, portalOpeningSlice.Value)
                );
            }

So, depending on the current platform the player is standing on, I set a "portal hint" on that same platform, on the correct hexagonal slice to guide them from the start platform to the bridge portal connecting the new game platform and the first gameplay zone.

Let's pause for a minute.

Last week I guided the player through the mirror portal which shows up when starting the game.

Now, I need to guide them through the "normal" portals, which require a gesture to be activated, meaning that I must finally make use of the gesture hint I implemented in week 32.

This means that I need a different behaviour for these hints, and rethinking about it, my naming of the hint mode (`EHintMode.portal`), last week, was wrong: that was the behaviour needed specifically for the mirror portal.

I'm going to refactor `EHintMode.portal` to `EHintMode.mirrorPortal`, updating all the references in last week code accordingly. Then, I'm going to re-introduce a "brand new" `EHintMode.portal` to use for the new behaviour.

I did the changes I mentioned, and some other minor fixes. Tomorrow I'll proceed with the logic of the new hint mode.

2025-01-21 - Portal opening hints

As the "normal" portals don't open automatically like the mirror portal, we need a slightly more complex FSM.

Let's go back to the predicates I used last week:

• `C` : player "captivated", meaning that the hint sphere is in their FOV

• `P`: player "positioned", meaning they're in the correct spot to open the portal

• `L`: player "looking" towards the portal (more precisely: properly rotated to open it)

• `T`: timeout, some specified time has passed

I'm adding a new one:

• `G`: opening gesture recognized

Now, to keeps thing manageable I'll try an approach where the hints are given in sequence, and if you fail to do a step after a certain timeout, the flow is sent back to the beginning. Unless one doesn't invalidate the previous predicates (for example: by moving from the spot), the flow should naturally go the "current" step (with no unwanted delays).

I implemented the new FSM and started testing it.

The core logic works pretty well, but there's a lot of work to be done before I can consider this part "complete".

There are still the glitches I briefly showed in the video at the end of the previous articles, and new ones related to transitioning to the `show_anim` state (that we didn't use with the mirror portal).

I'm going to try and fix everything in the remaining days of the week.

2025-01-22 - Hint particles appearance and disappearance

So, the two most unpleasant things in the current state of the hints are

• the "looking towards portal" check is not well tuned, meaning that it stops guiding the player gaze too soon

• the hint visuals (both "hint sphere" and "hint clip") appear/disappear abruptly

Let's face the appearance/disappearance problem first.

I'm going to add, at the state management level, two 0-1 values, `hintParticlesVisibility01` and `hintClipVisibility01`. I'm going to animate such values towards 0 or 1 depending on if these elements must be visible or not in a certain state.

Then, in the elements "presentation" scripts, I'm going to use these values to tune the placeholder VFX. I'm going to reuse the `VfxBhv` script I already used to handle the enemies and other elements, which easily allows me to change size, particles emission rate and colors. Good enough for now.

Ok, I did the `VfxBhv` refactoring and added a bit of logic to fade the visibility values, in the main smart particles state manager.

It doesn't look great, but it's far better than an instant visibility change.

Not worth a video, but I'll capture one tomorrow after fixing something else too.

2025-01-23 - Improving the gaze-guide

Today I'm going to work on improving the code used by the hint system to check if the player is properly rotated towards a portal.

The current check was built upon the one used to see if it was possible to open a portal. Why that's not good enough?

Well, up to now it's been possible to open portals backwards, or looking quite "sideways", as long as you are on the right half of the platform. If that's a good idea or not, is a topic for another day.

Right now, the point is that such check doesn't work well with the "gaze guide" hint: I want to check that the player is "precisely" looking towards the destination platform, before considering the suggestion by the hint system successful (and transitioning to the following state).

I implemented a simple check using the dot product between the direction towards the destination platform, and the player look direction projected on the ground.

When the dot product is close to one (currently, greater than `0.95f`), it's considered good.

It's not perfect, but I'd say the system starts to properly "guide" me to open the portal!

The `guide_look` fade out is pretty ugly, but at least is there.

Instead, the `guide_anim` clip showing the hand gesture still appears/disappears abruptly.

2025-01-24 - Fading the gesture hint

Today I have very little time available, but I should manage to do at least a little thing: remove the instant appearance/disappearance of the arms giving the hint about how to perform the portal opening gesture.

Ok, it was pretty quick: I just added a parameter to the "particles body" VFX to set the spawn rate of the particles, and hooked it up to the `hintClipVisibility01` I had already added a couple of days ago.

This is a test of the complete sequence, opening and crossing the three portals of the menu zone.

Next week I'm going to be super busy, so I'll probably be back with a new post in two weeks. Or three. But hopefully two. Stay tuned!

2025-01-06 - Limbo zone logic

Let's start from the beginning, the limbo platform where you need to enter Particular Reality and embody the avatar. I wrote (and sketched):

I need to push the player roughly near one edge of the hexagonal platform, and facing the opposite edge. This way, the portal to cross, which is going to "split" the platform in two halves when it appears, will open in front of the player, well visible.

Having some kind of arrow stamped on the platform would be effective, but too crude for my taste. What I'm thinking is having a dynamic hint system, based on floating particles, that goes into the player field of view and moves trying to push him into a certain spot.

From a high level perspective, the logic of the limbo zone should be:

• the zone is initialized with the mirror portal closed

• if the mirror portal is closed

  • If the player is not on the bottom half of the platform, facing towards the top half, bring them in that status with the hint system

  • otherwise, open the mirror portal

• if the mirror portal is opened

  • check the proximity of the player with its mirrored double, and if crosses a certain threshold, cross the mirror portal and embody the avatar (this part is already implemented, even if needs to be refactored)

  • optional: if a certain time has passed and the player is not approaching the portal, maybe use the hint system to encourage them to do that

According to my experience, the best way to think about gameplay logic is by having states activated by arbitrarily complex predicates.

I'm going to do a bit of refactoring, making it easier to check the predicates needed without duplicating the logic I already have to check if the player is in the correct position to open a portal.

I did the refactoring I wanted, or at least part of it.

I also wrote the "tentative" code for the limbo zone logic.
Without thinking too much about where this is currently placed, here's the snippet:

            if (!isPlayerReadyToOpenPortalTowards(Hex.EDir.N)) { 
                RTSpot mirrorPortalOpeningSpot = new RTSpot(
                    WorldStateManager.getStartupLoc(),
                    Hex.EDir.S
                );
                rSmartParticlesSM.setPortalHint(mirrorPortalOpeningSpot);                
            } else if (rPortalSM.getState() == Portal.EState.opened) {
                const float fCROSSPORTAL_HINT_WAIT_SECS = 5f;
                float fPortalBeenOpenedSecs = 
                    rPortalSM.getCurrStateElapsedTime();

                if (m_fTimeAtCurrLoc > fCROSSPORTAL_HINT_WAIT_SECS) {
                    RTSpot portalDestinationSpot = new RTSpot(
                        WorldStateManager.getMenuStartLoc(),
                        Hex.EDir.N
                    );
                    rSmartParticlesSM.setGotoSpotHint(portalDestinationSpot);
                }
                mirrorPortalHandling();
            }

The idea is that, at this level, I simply set what kind of hint I want depending on the situation.
Then, in the `SmartParticlesStateManager`, one layer below, I will appropriately use the features we implemented in the last weeks to guide the player.

I'm talking, of course, of the implementation of the `setPortalHint()` and `setGotoSpotHint()` methods, which are currently empty.

2025-01-07 - Improving visual debugging

I think that, before proceeding, it makes sense to revisit and improve the debug visualization features related to the player movement/target platform selection.

That way, I'll be able to easily understand if the conditions that should activate a certain hint are true at a specific time.

I haven't much time to write today, so I'm going to keep it brief.

I'm refactoring the old `DebugPlayerBhv` and a couple of other scripts so that they follow the pattern of the `DebugSmartParticlesBhv` I recently discussed. This avoid some code duplication and improves the organization of the scene, removing unnecessary interdependencies.

Things are gradually getting a bit tidier, and that's great, even if it's taking a bit more than expected. Can't escape Hofstadter's law.

I added a couple of timers indicating how long one has been on a certain platform, and on a specific hexagonal slice on that platform.

I also drafted a small panel following the player on the ground, showing the timer values and other information that might get useful when testing the hint system.

It's not worth a video, but here's an image.

2025-01-08 - Restructuring hint state management

I realized that to quickly test the hint visualization, in the last weeks, I cheated a bit and calculated the world transform for the hint sphere accessing directly the player tracking data from the Unity scene. That doesn't sit right. I must do the work and have the state management layer not dependant on the presentation layer. This calls for some more preliminary refactoring which won't carry visible results.

Ok, I was able to mostly fix what I wanted, but I had to do it in a hacky way.

To fix it properly, I will need to do some more in-depth restructuring.

Long story short, in the state management layer, at the moment, I only deal with zone-relative coordinates: while in the presentation layer, when two zones are loaded, I can do calculations involving platforms and elements from both zones, I can't really do that in the state management layer.

If I want to be able to have logic involving an adjacent zone, that you are maybe peeking into from an opened bridge portal, I will need to fix that.

For now, I did the barely minimum I needed to be able to have the `guide_move` state working when the destination spot is not only on the current zone (which was the "easy" case) but also on the single platform of the adjacent zone that is on the other end of the current bridge portal, if any.

As the "mirror" portal is a special case of bridge portal, connecting the limbo zone to the menu zone, this should allow me to implement the hint telling the player to move into the portal, if they don't figure that out after a while. I will just need to set the target spot for the `guide_move` state to the "other end" ot the portal.

2024-01-09 - Hint modes design

In the zone logic code written Monday, I called the `setPortalHint()` and `setGotoSpotHint()` stubs: it's time to fill them in, defining how the low level states of the smart particles subsystem (that we already implemented) are meant to be used to give players hints in specific contexts.

I'm going to introduce the notion of hint mode to describe the "high level" hints I want to have readily available to the zone logic code. Each hint mode will have its custom path in the smart particles FSM. Maybe, I will then factor this out to properly nested state machines, if it's worth it.

        public enum EHintMode {
            none,
            portal,
            gotoSpot
        }

So, let's define a simple (hopefully) FSM for each of these two modes we need for the limbo zone, and assume that calling `setPortalHint()` and `setGotoSpotHing()` activates each one of them.

Let's start from `EHintMode.portal`.

To be able to open a portal one must be

• in the correct spot

• facing the correct direction

So the hint particles should

• catch the player attention (`captivate`)

• if they are not at the right spot, send them there (`guide_move`)

• if they are at the right spot, but are not facing the correct direction, make them look that way (`guide_look`)

The system must be robust, meaning that if after some time the player fails to follow the hint, it should go back to `captivate` their attention and try again.

To make the diagram readable, let's assign some one letter identifiers to the predicates we are going to need:

• `C` : player "captivated", meaning that the hint sphere is in their FOV

• `P`: player "positioned", meaning they're in the correct spot to open the portal

• `L`: player "looking" towards the portal (more precisely: properly rotated to open it)

• `T`: timeout, some specified time has passed

I added a `done` state just to have the diagram flowing better. And we're going to add some extra state like that to take care of the appearance/disappearance of the hint particles anyway.

Let's move to the `EHintMode.gotoSpot` FSM, which should be a bit simpler.

To send the player to a specific spot, we shouldn't need `guide_look`: we just catch their attention with `captivate` and then use `guide_move`. If the player fails to reach the destination spot in a certain time, and they're not looking at the hint sphere (meaning that the player attention is not captivated), we go back to `captivate`, repeating the loop until the player reaches the spot.

Tomorrow I'll try to implement and test all this.

2024-01-10 - Hint modes implementation

As first step, I'm going to implement some utility functions to evaluate the predicates I described yesterday, which will probably be the most delicate part.

Then, I will map the logic defined by the diagrams to the code, which should be straightforward.

Ok, my time is over for now, but let's say that I implemented what I designed yesterday, and it somewhat works.

There's plenty of details to take care of, and polishing to be made, but for this week, this is it:

The worst thing is how the hint abruptly disappears as soon as I'm in the correct direction to open the portal. Then, it sometimes intersects platforms, and it's not clear where does it come from when, after opening the portal, it suggests go cross it by guiding to the target platform.

Well, you gotta start somewhere!

• `show_anim`: the particles should morph to a detached body (or maybe just hands?) which playbacks a captured animation, inviting the player to emulate it

  • input: where to perform the animation, and what animation to perform

Let's go!

2024-12-30 - Revisiting motion clip management

So, what's the motion clip capture and playback situation?

If you want the full story, you need to read:

• Deep dive into body tracking - Tales from the Meta SDK

• AP Week 16: Motion capture and playback

• AP Week 17: Motion clips management

• AP Week 18: Additional clip management and debugging

• AP Week 19: Clip playback on skinned meshes

Otherwise, here's a quick summary.

I can, using the in-VR "clip management" UI I did:

• capture body tracking data and save it to a motion clip

• play a saved clip, optionally changing its playback speed and looping behaviour

• define a selection (setting begin/end frames) on a clip and, if needed, extract it to a new clip

The idea is saving clips for the gestures I want to teach, giving them identifiers, and playing them back with a suitable skinned mesh character which is going to be part of the smart particles subsystem.

Until now, I just saved clips and played them back at the same position they had been captured, and using a specific test character (currently, the magenta particles avatar seen in week 19).

Now, I need to improve the clip playback system so that I can

• play clips on different (compatible) targets

• play clips at different positions/rotations and not only where they were captured

I revisited the code I wrote months ago and did a bit of refactoring. Now, the `UpperBodyClipPlayerBhv` doesn't directly apply the clip data to the test character: it applies such data to any linked node having a script inheriting from a new abstract class I defined, `AUpperBodyClipPlayerTargetBhv`.

The logic which was embedded in `UpperBodyClipPlayerBhv` got factored out to a `UpperBodyClipTestPlayerTargetBhv`, but now I will be able to reuse `UpperBodyClipPlayerBhv` to drive other setups.

Just a bit of ground work for today!

2024-12-31 - Capturing and loading hint clips

To get started, I'm going to capture a couple of motion clips showing how to open the portal and walk into it.

I'm going to define an enumeration identifying this set of motion clips.

public enum EHintClipId {
    openPortal,
    stepIntoPortal
}

Then, I'm going to capture two clips (placeholders of course, without caring that much about capturing them nicely) for these hints.

I did a single capture of opening and entering a portal, and then extracted the two clips from there:

Currently, the motion clips are stored as simple files on mass storage. That means that I would need to "manually" copy such files on the Quest file system if I want to to test a standalone build. Embedding the data in the build would be preferable.

Actually, I already solved the same problem when dealing with the level descriptions: I don't remember if I discussed the details on the DevLog, but I added a secondary data path which allows to load them from the Resources data and not only from external files.

I'm going to extend such simple system to also work with motion clips.

Done! From a quick test, the loading seems working. Tomorrow I'll setup a skinned character to be used by the hint system and play the clips on that,

2025-01-01 - New year, new skinned character

In Italy they say that what you do on the first day of the year, you’ll do all year around.

Guess what, it doesn't really work, but still, let's pretend it does and do a little work today, hoping that I'll manage to keep developing the game without too many forced breaks during the whole new year.

So, I duplicated the character setup I'm currently targeting when doing playback from the clip management UI. I changed the particles color to the same yellowish tint I configured for the placeholder VFX.

I also created another `UpperBodyClipPlayerBhv` and linked it to the new character setup.

Then, I added another subsystem to the smart particles one, naming it hint clips, which drives this hint clip player targeting the hint character. It's not that confusing.

I tested the new setup, and it works like expected.

Before working on the placement of the clip, I need to deal with the problem that I don't want the hint character to feature the full upper body, but just the arms and hands.

You might remember that to hide the legs I did a quite ugly hack: I forced the scale of the legs bones to a very small number in `LateUpdate()`. That wasn't perfect, because there's still two shiny points at the base of the legs that show up, bud I was happy enough of the solution, considering it's just temporary.

This time, unfortunately, I can't use the same trick, because this is the bones hierarchy:

I can scale down the legs and the head (scaling down the `Neck` node), but I can't scale down the chest without also affecting the arms, under the shoulders.

So, what can I do?

I went out of my comfort zone and tried using Blender to actually make an alternate version of the character FBX where I stripped the meshes of the body parts that I didn't want (keeping the skeleton untouched).

I managed to have an "arms and hands only" version of the character, and it rendered properly, but the arms and hands didn't animate well. What was wrong? I know very little about modelling and rigging, so I really don't know. But to understand if it was because of the things I had removed, I tried just importing and re-exporting the character FBX, with no modification, and that didn't work either. Is some obscure export setting configured the wrong way? A rabbit hole not worth getting into.

Time for a plan B...

Opening the model in Blender wasn't totally useless: it made me remember that the character mesh actually features three submeshes. I should have known, because I remembered dealing with three different materials attached, and AFAIK you can't have different materials without different submeshes.

To see the submeshes in the inspector, you need to select the mesh in the project tree, inside the imported FBX tree. The entry with the little cyan grid icon. In this case, I got this:

Ha! The arms are indeed a specific submesh, the blueish #1.

Let's try something...

Well, it worked. Here's a clip.

Tomorrow I'll clean up a bit the ugly hack I did, and explain what I did.

2024-01-02 - Removing submeshes

So, what did I do to only render the arms and hands of the character skinned mesh?

After realizing that what I wanted to keep was in a separated submesh, I tried to generate at runtime a new `Mesh` object, cloning the original, except for the submeshes definition, where I only kept the one I wanted.

A submesh is made of a sequence of triangles, and each triangle is defined by the indices of the three vertices that form that triangle (taken from the `vertices` array).

When you create a new `Mesh` object via code, you can specify the number of submeshes, and then the Unity `Mesh` API lets you get/set this triangles definition for each submesh via these methods:

• `int[] GetTriangles(int submesh)`

• `void SetTriangles(int[] triangles, int submesh)`

So, my hack leaves all the "base" mesh data unaltered (changing it but keeping it consistent might be complicated), without removing the vertices, normals etc, but changes the submeshes definition so that Unity only uses a subset of the data when rendering.

Not optimal in terms of memory use, but perfect for my prototyping purposes, as there are no visible artefacts.

And there's other good news: I can use the same approach for the other characters, where I hid the legs by scaling their bones hierarchy to `0.001` in `LateUpdate()`. Less useless work done, and no visual artefacts (the two shiny dots at the base of the invisible legs I have right now).

Refactoring and clean-up time!

This is the method I added to my character presentation script:

private static Mesh PartialClone(
        Mesh rMesh, 
        int[] rSubmeshesToKeep) {
    Mesh rNewMesh = new Mesh {
        subMeshCount = rSubmeshesToKeep.Length
    };
    rNewMesh.vertices = (Vector3[])rMesh.vertices.Clone();
    rNewMesh.normals = (Vector3[])rMesh.normals.Clone();
    rNewMesh.uv = (Vector2[])rMesh.uv.Clone();
    rNewMesh.colors = (Color[])rMesh.colors.Clone();
    rNewMesh.tangents = (Vector4[])rMesh.tangents.Clone();
    rNewMesh.boneWeights = (BoneWeight[])rMesh.boneWeights.Clone();
    rNewMesh.bindposes = (Matrix4x4[])rMesh.bindposes.Clone();

    int iNewMeshSubmeshesIdx = 0;
    foreach (int iSubmeshToKeepIdx in rSubmeshesToKeep) {
        int[] rTriangles =
            (int[]) rMesh.GetTriangles(iSubmeshToKeepIdx).Clone();
        rNewMesh.SetTriangles(rTriangles, iNewMeshSubmeshesIdx);
        ++iNewMeshSubmeshesIdx;
    }

    rNewMesh.RecalculateBounds();
    rNewMesh.RecalculateNormals();
    return rNewMesh;
}

I didn't check if I could avoid cloning the mesh data (I probably could), but let's stay on the safe side, considering it's just a temporary fix for a temporary character mesh.

Then, I exposed a `m_submeshesSetup` setting to the inspector, to configure what kind of visualization I want for each character instance:

public enum ESubmeshesSetup {
    FullBody,
    UpperBody,
    ArmsOnly
}

[SerializeField] private ESubmeshesSetup m_submeshesSetup;

Finally, in `Awake()`, I called the `PartialClone` method indicating what submeshes to keep depending on the configured value.

const int iBODY_SUBMESH_IDX = 0;
const int iARMS_SUBMESH_IDX = 1;
const int iLEGS_SUBMESH_IDX = 2;

protected virtual void Awake() {
    if (m_submeshSetup == ESubmeshSetup.UpperBody) {
        m_rSkinnedMeshRenderer.sharedMesh = PartialClone(
            m_rSkinnedMeshRenderer.sharedMesh,
            new int[] { iBODY_SUBMESH_IDX, iARMS_SUBMESH_IDX }
        );
    } else if (m_submeshSetup == ESubmeshSetup.ArmsOnly) {
        m_rSkinnedMeshRenderer.sharedMesh = PartialClone(
            m_rSkinnedMeshRenderer.sharedMesh,
            new int[] { iARMS_SUBMESH_IDX }
        );
    } // else do nothing and keep the full body mesh
}

Sometimes, you solve a basic problem in a stupid-simple way, and then are presented with a more complex version of the same problem which pushes you to find a less naïve solution. If feels nice when that new solution also provides a better solution for the original, basic problem. That's what happened this time, with the removal of the legs scaling hack and its artefacts.

2024-01-03 - Placing hint clips

So, we have a way to load, identify and play on a special "arms only" character some motion clips.

It's time to properly place them in the game world.

Honestly, I'm not sure if I want to show them

• into "ideal" positions, like relative to a spot (as we defined it in week 31, so a platform location and a platform slice identifier)

• according to the current player position and orientation, so that they can basically "match" the hint clip with their hands

There's another problem to take into account: players have a different stature.

Ideally, I would want to "transpose" a captured animation to the hint character but using the bones size derived from the live tracking data, so that the animation adapts to the actual player stature. This feels a bit overkill, especially considering the current stage of development, but I wrote down a note about looking into it at some point in my "polishing / improvements" notes.

Instead, to keep it simple, I will just scale the hint character a bit according to the detected player height and - as I'm doing the captures - mine (`1.75m`).

For example, if I detect that the player height is `1.90m`, I will set the hint character scaling to `1.90 / 1.75 = 1.085`, if it's `1.60m` the scale will be instead `1.60 / 1.75 = 0.914`.

This is a common approach in XR development, and if you're surprised that a uniform scaling can work pretty well, consider that the width of the arm span of a person is roughly the same as their height, as shown by the square in the Vitruvian Man.

Coding time!

I proceeded without significant problems. For now, I decided to use the spot identification system to place the hints character. I'll probably add other options along the way.

I wrote a bit of code to take care of the character scaling depending on the player height, and tested that it affects the character correctly. That said, the feature is incomplete, because I don't have a calibration sequence yet, so I don't have the correct value for the player height. I tried calculating a kind of "continuously adjusting average height", but I didn't like the result, so I scrapped it. Instead, I wrote a nice `// TODO` into the method implementation to calculate the scale, and had it return `1.0f`.

Here's a short clip where I test the placement of the hint animation via inspector:

And week 32 is done too. I'm now able to show hint clips with adequate in-game placement.

I should finally have all the pieces I need to implement the tutorial sequence I described in week 25... so, don't miss the next article!

Let's see if I manage to do a bit of extra work in these XMAS holidays, and make up for the lost time.

2024-12-23 - Hint particles setup

So, what was I doing before this forced break?

I started working on the smart particles, doing a bit of design and then introducing a subset of them called probe particles which I linked to the portal opening/closing mechanic.

That last part wasn't completely done, but before going back at it I prefer to draft the other smart particles behaviours.

Let's copy the definitions of a subset of smart particles states from the design done during week 29:

• `captivate`: the particles should catch the player's eye. A basic way to do it is move towards the centre of the player's field of view, at a certain distance, and idle around there.

  • input: the player state data

• `guide_move`: the particles should guide the player towards a certain spot.

  • input: the player state data and the destination spot (a location and a hexagonal slice identifier should suffice)

• `guide_look`: the particles should invite the player to look towards a certain point

  • input: the player state data and the destination point

These states sound quite interconnected, because they deal with particles that catch the player attention and guiding them.

So, I'm going to introduce another subset of smart particles, called hint particles, that will take care of these states, similarly to how the probe particles dealt with the portal interactions.

To guide the player towards a specific spot, or to have them looking towards a certain point, I need to get their attention first. That's the idea behind the `captivate` state.

But thinking about it, I'm not sure I need an explicit state for that. Maybe, I could implement it as part of the `guide_move` and `guide_look` states, with a nested state machine which makes sure that, before guiding the movement or the game of the player, the hint particles have the attention of the player.

Anyway, I'm going to keep the state separated for now, which also facilitates testing and keeps things simple.

Let's say that, one way or the other, the hint particles have a "core", which is a sphere that acts as placeholder volume for the particles. We can assume that, if such sphere is in the player field of view, they're seeing the particles too.

I'm going to put a sphere in the scene, and try to steer it in front of the player. I also want to add a check to see if the sphere is actually in the camera frustum at a certain instant.

Done! Here's the result, while testing in-editor:

Notice that when the sphere is totally visible, is green, when it's partially visible, it's yellow, and when it's completely outside the frustum, is red.

Here's how I implemented a simple check using the camera frustum planes and the sphere center and radius:

public enum EFrustumCheckRes {
    totallyInside,
    totallyOutside,
    partiallyInside
}

public static EFrustumCheckRes SphereFrustumCheck(
        Camera rCamera, 
        Vector3 vSphereCenter, 
        float fSphereRadius) {

    bool bIsTotallyInside = true;

    Plane[] planes = GeometryUtility.CalculateFrustumPlanes(rCamera);
    foreach (Plane plane in planes) {
        float fDist = plane.GetDistanceToPoint(vSphereCenter);

        if (fDist < -fSphereRadius) {                    
            return EFrustumCheckRes.totallyOutside;
        }

        if (fDist < fSphereRadius) {
            bIsTotallyInside = false;
        }
    }

    return bIsTotallyInside ? 
        EFrustumCheckRes.totallyInside : 
        EFrustumCheckRes.partiallyInside;
}

The floating behaviour steers the sphere towards a point at two meters from the player, along their gaze vector. Notice that the sphere goes up and down too, following the "look" direction.

Basic but effective. The only problem is that, looking down, I can send the sphere below the platforms, which doesn't make sense.

I'll fix it later projecting to the ground plane when under a minimum height.

2024-12-24 - Guiding towards a specific spot

Today, I'm going to take care of the `guide_move` state:

• `guide_move`: the particles should guide the player towards a certain spot.

A long time ago, I defined `RTLocation`, a pair of identifiers (zone id and platform id).

Then, I have the `Hex.EDir` enumeration, which identifies each of the six slices of an hexagon:

    public struct Hex : System.IEquatable<Hex> {
    // ...
        public enum EDir {
            NE,
            N,
            NW,
            SW,
            S,
            SE,
        };
    // ...
    }

I think that, in terms of "guiding the player somewhere", I won't need to be more specific than "that slice of that platform". Additionally, I remember passing as parameter an `Hex.EDir` together with a `RTLocation` a few times already.

This tells me that it's a good idea to introduce a new simple type to indicate a specific platform slice. I could add an optional `Hex.EDir` to `RTLocation`, but that could get confusing.

So, let's define `RTSpot` (not much imagination with naming, today) pairing an `RTLocation` and a `Hex.EDir`.

Quick and easy. For testing/debugging purposes, another thing I often wanted is having the platform slices labelled, so that I can quickly check that things are correctly positioned.

Let's take care of that too.

Platform labelling done! While I was at it, I not only labelled the edges but the platforms themselves, using the string identifiers I set in the level definitions. This way, I will be able to easily recognize the "meaningful" platforms while testing.

Now it's time to add the logic to send the hint sphere towards a specific spot. Should be easy enough.

I added the zone and platform numeric identifiers to the platform label, so I can easily know what values I need to insert in the testing script if I want to send the sphere to a specific spot.

This will need a bit of tuning, but the essence is there!

2024-12-25 - Guiding the player gaze

Hey it's Christmas! Maybe I shouldn't work... but come on, just a little!

So, while to guide the player towards a specific spot I just sent the hint sphere there, just above platform height, I imagine guiding their gaze will be a little more complicated.

While in the `captivate` state the hint sphere goes to the middle of the player field of view, in this case I should try to send it to the edge of that field of view, suggesting the player to look that way.

You are probably familiar with this kind of visual hint if you have played any 3D space sim: you get some tiny arrows at the sides of the screen, pointing at enemy ships outside the field of view. If you rotate towards one of these arrows, you end up getting the enemy ship it was pointing at in front of you.

My case is slightly different, because I don't have to position UI elements at the edge of the screen, but the hint sphere in 3D space. Still, the core idea is the same.

I'm not sure if I'll ever need to have the player look up or down, that will be influenced by how much I will go crazy with the level design.

To limit the scope, I'm considering planar levels, so the hint sphere should basically push the player to rotate left or right (whichever is faster).

For example, let's assume the player head is vertical, and they're looking towards North, but the current hint is to look towards South-West: the hint sphere should go to the left edge of the player field of view until they're rotated to look towards SW (and the sphere would be at the center of the FOV).

I can imagine different ways to implement this - let's try the simplest one.

I tried implementing the straightforward approach I had in mind, but it doesn't work well. Bug or fundamentally wrong idea? I'll find out tomorrow...

2024-12-26 - Fixing the gaze hint

If you've been reading the DevLog for a while, you already know what I'm going to do now: add a bit of debug visualization to understand what's wrong.

I added a couple of extra spheres for debugging purposes: one to indicate the point we want the player to look at (the hint target), and another to indicate the position where we want the hint sphere to go, at the edge of the field of view.

I did an even simpler version of yesterday's code, which only considers the horizontal angle between the player forward direction and the target direction, because I felt that the error was in the calculation of the vertical rotation.

The resulting code worked pretty well, as can be seen in the following video:

The blue sphere is the target point, and the sepia sphere is the hint point, which the usual hint sphere continuously tries to reach. Notice that the sepia sphere sticks to one side of the frustum when the target sphere is outside of the field of view, and lies on the invisible line between player and blue sphere instead.

So how does this work?

• I get the forward vector of the player and the head position

• I calculate the target direction, which is the vector from the player head to the target point, normalized

• I calculate the signed angle between these two direction vectors, which tells me the "yaw" needed to rotate the head towards the target

• I clamp this needed rotation angle to half the horizontal field of view

• I rotate the forward vector horizontally by the clamped rotation angle

• Finally, I calculate the point along the "rotated forward" at the desired distance from the head

Here's the relevant code snippet:

        public static Vector3 getGuideLookPos(Data rData) {

            Vector3 vHeadPos = BodyInputManagerBhv.i().getHeadPose().position;
            Vector3 vFwd = BodyInputManagerBhv.i().getPlayerFwd();
            Vector3 vUp = Vector3.up;

            Vector3 vTargetDir = (rData.m_lookAtPoint - vHeadPos).normalized;

            float fFwdToTargetAngle = 
                Vector3.SignedAngle(vFwd, vTargetDir, vUp); 

            float fHalfHFOV = CamerasManagerBhv.i().getHorizFOV() * 0.5f;             
            float fLookToTargetClampedAngle =
                Mathf.Clamp(fFwdToTargetAngle, -fHalfHFOV, fHalfHFOV);
                        
            Quaternion qHorizRot =
                Quaternion.AngleAxis(fLookToTargetClampedAngle, vUp);            
            Vector3 vHintDir = qHorizRot * vFwd;                        
            Vector3 vHintPos = vHeadPos + vHintDir * Params.fLOOKAT_DIST;
            
            return vHintPos;
        }

There are probably better solutions to this kind of problem, the one I implemented is essentially a 2D solution, but considering the initial assumption of planar levels, it should work pretty well.

Maybe I'll refine it or change it in the future!

2024-12-27 - Hint VFX placeholder

I'm tempted to draft the last remaining state of the smart particles, the `show_anim` state which is supposed to teach the player a gesture by displaying an animation to perform.

But it's been quite some time since I implemented the core features I'm going to need (it's a good moment to go back and read the posts for weeks 16, 17, 18 and 19), and I feel like it's going to take more than one day to revisit all that stuff and make good use of it.

So, considering it's Friday, I'm going to postpone that to next week. So, what I can do instead?

It feels a good moment to do a bit of clean-up and refactoring of the work done in the past days.

I'm going to add a placeholder VFX for the hint particles, and relegate the spheres I've been showing so far to the debug mode.

Ok, that went pretty well.

I put a placeholder VFX (basically the one I did for the fireball a long time ago, but set to yellow) in place of the hint sphere I've been using so far.

The visualization that was in place until now was relegated to the debug mode, so if I sense that something is off with the VFX placement, I can easily look behind the scenes.

While I was at it, I drafted a generic script for the "debug only" visualization of a subsystem, factoring out some logic that until now I had repeated multiple times in other debug behaviours.

I will progressively move to this new system other debug components. By isolating these elements, I will be able to easily strip them out from a build.

In the Unity scene, I have a separated subtree for the debug nodes. The smart particles debug node has a reference to the "non-debug" manager of that subsystem, which uses as its data source. Only when the debug mode is active, it takes care of providing an alternate visualization (in this case, placing the three spheres we've used yesterday).

Week 31: done!

Unity 6 was officially released, and that prompted me to update my assets on the store to make them compatible with it, something I had been procrastinating for a while. Additionally, I had to switch to full-time contract work for a while to meet a deadline.

Well, time to get back on track!

2024-11-11 - Particular portals

Last time, while I was wrapping up the post, I mentioned that I had a new idea about a possible relationship between smart particles and portals.

So, here it is.

Let's say that it's the particles that give powers to the basic player skeleton you embody at the beginning. Then it makes sense that it's the particles which open the portal, flowing from the body to the portal, leaving a "bare" skeleton. If you close the portal, they flow back to the body. If you cross the portal, they stick back to you while you cross it.

I think it makes sense to make the "smart particles" a central element of the game, something which connects the player to the alternate reality of the game world.

Before doing that, I'm going to do a little design.

Let's revisit the portal FSM:

I think that we can mirror these states with parallel states of our smart particles FSM. The only additional complexity should be that they should also let the player understand what platform is the current target, acting like a selector. And there might no be a valid target, depending on the level layout and on the player position on the current platform.

I'm going to call these particles acting as a selector probe particles, and I think it makes sense to have them work independently

I might be wrong, but I feel like the probe particles should be in one of three states:

• `invisible`: for the states where the player is not embodying particles at all

• `idle`: for the cases where when there's no valid target platform - they should "stay back" near the player, in their field of view, ready to jump to select a nearby platform

• `follow_portal`: for the cases when there's a valid target platform -they should float there, and act accordingly to the portal state

  • `closed` -> idle, ready to open the portal

  • `opening` -> opening the portal and bringing it to the player

  • `closing` -> closing the portal/sending it back

  • `opened` -> idle, on the opened portal

By deferring to the portal state, I should be able to avoid replicating the same logic: basically, the `follow_portal` state tells the probe particles to "do what the portal does", which makes sense as they must be synchronized.

And the impact of the `follow_portal` state is not limited to the probe particles, because according to the idea I described at the beginning of the post I'm imagining that, when opening a portal, the particles that normally follow the skeleton flow to open the portal, and stick back to the player body when they cross the portal.

Let's recap:

• I need some probe particles, still managed by the smart particles subsystem, but that basically act as target platform selector

• when a target platform is selected, I can probably drive the whole subsystem on the basis of the portal state.

  • `closed`: probe particles idle, highlighting the portal

  • `opening`: probe particles "open" the portal together with the body particles which are flowing there

  • `closing`: the body particles flow back, the probe particles go back to idle

  • `opened`: all the particles are keeping the portal active. If the player crosses the portal, the body particles stick back to the player, and the probe particles go back to probing for the next destination

I'm definitely getting some details wrong, and that's why, as usual, tomorrow I'm going to start prototyping and figure out the details along the way.

2024-11-12 - Probe particles

I'm going to use a few floating balls as placeholders for the probe particles that I want to procedurally animate.

Time to write some code!

Ok, I got to something which is "good enough" for now.

I instantiated six spheres and wrote a bit of code which animates them using a simple mix of attraction/repulsion forces, with some bounds, randomness and sin waves oscillation thrown in.

I can specify a target point which attracts them, which will allow me to direct the particles towards a specific platform.

This is me manually moving the target point with the mouse in the Unity scene view, after a bit of tuning of the parameters involved.

2024-10-13 - Probing platforms

Now that I have a bunch of floating spheres I can send around, I need to send them at the right place.

As first step, depending on if there's a valid target platform or not, I'm going to set as attractor point for the probe particles

• the center of the target platform

• the point at the vertical center of the player field of view, at a certain distance

As height, I might go for something like 1.5 mt, or maybe the head height.

I can infer state from the player and portal state management, so, at least for now, I'm going to work directly at the presentation layer, and drive the probe particles from the `SmartParticlesBhv` script.

Well, it was quick, but there's definitely a lot more tuning to do.

Trying with the headset, the motion feel too wide and chaotic.

The little time I had available for today ran out, so I'm going to get back at it in the future.

2024-10-14 - Portal preopening and postclosing

I've been doing some thinking, and to have things flowing the way I'm imagining them, I think I need to introduce a small intermediate step in the portal opening.

Now, when the player performs the gesture, the portal instantly shows up and comes forward. I want everything pretty fast, to facilitate frenetic action gameplay, but I also want everything to be "readable" and flowing seamlessly. When the portal didn't move, but directly appeared on the player platform, just doing the portal pop-up animation was maybe fine. Now that it opens on the target platform, and then comes forward on the current platform, well, it's not great.

So, I'm going to introduce a separate state where the portal "opens" on the target platform, and only after that phase it will come forward to the player.

This small delay at the opening should offer an opportunity to make use of the probe particles (which will in someway create the "border" of the portal), and will also filter out some brief but unwanted portal activations, if the gesture gets detected for an instant but it wasn't maybe intentional. Now, you will have to keep doing it for a short while before the portal starts coming towards you.

Let's get going!

I feel a bit lazy today, so instead of renaming the old `opening` state to `comingForward`, and introducing a new `opening`, I just added a `preopening` state and plugged it between `closed` and `opening` . I'll regret it at some point, of course.

I also added a `postclosing` state, to keep things symmetrical.

If you remember, I made sure that opening and closing the portal were reversible actions, so that you can abort or resume opening or closing a portal without ugly jumps in the flow. I did the same with `preopening` and `postclosing`, using a new "level" value going between 0-1, which goes up when `preopening` and down when `postclosing`.

I will skip the diagram of the revised FSM, as it's not that interesting and I'd like to focus on implementing the changes

I was going to test my changes, and I realized that I have never implemented a `TestPortalBhv` script as I'm doing lately. I guess it's a good moment as any to take care of that.

Ok, I added the new state, prepared the testing script, and also changed a bit the procedural animation. Not worth a video, but I'll try to make one tomorrow!

2024-10-15 - Binding probes and portals

I'm going to start binding the renewed portal logic (featuring `preopening` and `postclosing`states) to the probe particles behaviour.

A couple of days ago, I tested "sending" the particles to the current destination platform. Now, I want to do something with them when I activate the portal.

For example, let's move the probe particles to the six vertices of the portal hexagon, and have the edge gradually fade in. The idea should be that the probe particles, sent forwards to the target platform, "cut" the space and bring it towards the player.

In the following video, there's a significant slow down factor modulating the portal animations. I usually use this trick to make sure the animation logic is sound and flows without jumps, which might be hard at full speed. When everything is convincing and stable at slow speed, I remove such factor.

Ok, it's not great, because the motion of the probes towards the vertices is too unpredictable. I need to have the spheres reach the vertices in a fixed amount of time. I made the portal edge additive and fading (until now, it was solid), a tiny improvement to avoid it appearing and disappearing instantly.

There's still a lot left undone, but I laid out the foundations: I added the needed transition states to the portal logic (doing a bit of refactoring on the go), and linked the probe particles state management to the portal, so that I can properly animate them.

The next steps are obvious. Unfortunately, I already know that I won't able to work on the project next week. Fingers crossed for the week after that!

The smart particles should be one of the two subsystems related to the player avatar (the other is the "skeleton" management discussed in the week 27 and week 28 articles).

They should act as both an embodiment of the player when in active gameplay, and as a hint system teaching him gestures and other gameplay actions.

At least, this is what I'm imagining at the moment: the idea is to start somewhere and extend the usage as the gameplay needs change.

So where do we start? Let's extract from the design done in week 25 a list of things that the smart particles should be able to do:

• catch the attention of the player by going into their field of view

• guide the player towards a specific spot

• show/suggest a gesture to perform

• attach to the skeleton avatar and "follow" it

• detach from the skeleton avatar

Ideally, the smart particles should transition smoothly from one activity to another, morphing between shapes and effects as needed.

But that's for later, when I'm going to work on presentation and VFX. As usual, instead, I'm going to focus on the logic first, and just implement a quick placeholder visualization.

I'm still prototyping, after all.

I'm going to define some state identifiers, with a brief description of the logic that the state should implement and the input it would need.

• `captivate`: the particles should catch the player's eye. A basic way to do it is move towards the centre of the player's field of view, at a certain distance, and idle around there.

  • input: the player state data

• `guide_move`: the particles should guide the player towards a certain spot.

  • input: the player state data and the destination spot (a location and a hexagonal slice identifier should suffice)

• `guide_look`: the particles should invite the player to look towards a certain point

  • input: the player state data and the destination point

• `show_anim`: the particles should morph to a detached body (or maybe just hands?) which playbacks a captured animation, inviting the player to emulate it

  • input: where to perform the animation, and what animation to perform

• `skeleton_follow`: the particles follow the skeleton avatar, which should be the "normal" state during active gameplay

  • input: the player body data needed to match/follow it

• `skeleton_idle`: the particles don't follow the skeleton avatar, and stay idle wherever they were when they entered the state

  • input: probably none

• `invisible`: the particles are not shown (startup state)

From what I described, this subsystem sounds rather complex compared to other things I've implemented so far.

When something is complex, one should try to further divide into manageable subtasks, which can be reasoned about (and maybe implemented) independently. And in fact, each of these states might end up having its own nested state machine.

One step at a time, I'll get there.

2024-10-22 - Adding a new subsystem

As discussed, the smart particles are an important and quite complex element, so I'm going to introduce a new subsystem to take care of them.

This means, as usual, defining their state management and presentation code. The first step is drafting these elements with empty stubs that I will fill along the way.

At what "level" should the smart particles operate and, consequently, where should I put their state? Application or game?

Like the avatar state, I'm pretty confident they should go in the application state, as I need them working properly outside of actual gameplay, in the limbo/menu zones too.

I'm not sure yet if I'm going to need some kind of "gameplay only" data override to handle them in-game, but I'll figure that out in the future.

Ok then, I defined

• `SmartParticles` and `SmartParticlesManager` (state management)

• `SmartParticlesBhv` (presentation)

• `TestSmartParticlesBhv` (testing)

...and I wired things up so that the `SmartParticlesManager` is nested inside of `AppStateManager`.

This initial setup should be ok: I checked that the game runs and that all data is initialized properly.

The new subsystem currently does nothing, but also doing nothing is something, in the sense that it's not crashing the game.

2024-10-23 - Energy bars refactoring

Yesterday I discussed how the smart particles data had a bigger scope that the gameplay data, because they also need to run outside of gameplay.

But there's a little thing that bothers me, and it's the energy bars, for which the opposite is true. They are now always visible, while they should be a "gameplay only" feature.

Let's look again at our table about player management:

My initial idea was that health and mana should be represented through the particles following the skeleton.

I'm not sure exactly how that is going to happen, in terms of visualization, but in terms of structure and update logic, moving the energy bars management to the smart particles code I introduced yesterday sounds like a small step in the right direction. Or, at least, in a less wrong direction.

So, as a first step in implementing the smart particles subsystem, I'm going to move to `SmartParticlesBhv` (taking care of presentation) the references to the energy bar nodes and their management.

I will make it so they are only visible in the `skeleton_follow` and `skeleton_idle` states, where they should behave like the particles: following the avatar, or standing still.

To be able to test this, I will also implement a restricted version of the `SmartParticlesManager` FSM, limited to these states and inputs:

Of course, to be able to test it, I will also add the `startFollowingSkel` and `stopFollowingSkel` input signals as buttons of the `TestSmartParticlesBhv` script.

Ok, I did what planned. The only problem is that there's a mismatch between the hand pose that I'm using to position the energy bars and the upper body skeleton. I think it's the usual mismatch that was introduced at some point updating the Movement SDK, and that I already discussed a few times.

I have two options: I change the positioning of the energy bar to use the (slightly wrong) bone pose as taken from the skeleton, or I try to fix the mismatch once and for all.

Well, today I won't do either: it's getting late.

2024-10-24 - Attached and detached particles

The most obvious step to do today is adding back the "particles avatar" I recently disabled, which in my vision is also part of the smart particles system, and specifically of the two states that I decided to use as a starting point: `skeleton_follow` and `skeleton_idle`.

What I need to do should be pretty simple: give control of that "particles avatar" element to the script orchestrating the smart particles presentation (`SmartParticlesBhv`) and

• setting it hidden in the `invisible` state

• update it so that it follows the player in the `skeleton_follow` state

• keeping it visible, but not updating it with body tracking data, when in the `skeleton_idle` state

Well, that was quick as expected, for once! And not interesting enough to capture a video, but here's an image.

Obviously, the particles intersect with the skeleton mesh, and the result is an ugly mess.

But that doesn't bother me right now - it's still just a placeholder visualization.

Instead, there are two more significant things I want to work on.

For starters, I currently have more than one `OVRBody` script in the scene, because the quick thing to do, starting from the initial sample, was fetching the body tracking data that way.

But I did significant work (weeks 16-17-18-19 and the extra article "Deep dive into body tracking") to be able to drive a skinned character mesh from body tracking data saved in a custom motion clip, and I took care of that adding an indirection layer between sourcing the body tracking data and animating the skinned mesh.

Then, I only used that data path for playing saved motion clip, but I could do the same with live data. Why bother?

Well, as I'm starting to have multiple skinned meshed driven by the same tracking data, I can easily avoid some useless data duplication and processing. But a possible improvement in performances is not the primary factor: it's more about decoupling subsystems and keeping the data flow tidy.

It makes sense to start separating and keeping minimal the parts that take input data, and that use a specific API to do so. Today such data comes from the `OVRBody` script, tomorrow it might be something provided by Pico, Sony, Apple, Valve...

I know that I'm just prototyping, but I guess today I feel positive about the distant future.

Done! Now I have a single instance of the body tracking data, and I use that to control two separate skinned character meshes, the "skeleton" and the "particles".

And I have an idea for a third one, but we'll discuss that tomorrow.

What's the second think I wanted to fix?

There's a problem with the "particles" character that I've been ignoring so far: sometimes, when I move/rotate my head while testing, the VFX particles disappear for no apparent reason. When that happens, moving back causes them to become visible again.

Let's see if I manage to remove this annoying glitch.

Ok, after a brief search, I found about a culling parameter that one can set in the VFX graph to avoid processing "out of bounds" particles.

There were some fixed bounds set, centered on the origin of the world. So, when I moved outside of those bounds, the particles disappeared.

To try a fix, I changed the center of the bounds so that it follows the main camera.

Well, it works. It's not great because I should pass in as parameter the position to use as center, and not make it dependent on the camera. But it's super late and I'm happy enough of having found the cause of the problem - a better fix won't take much time.

2024-10-25 - Yet another character instance

While implementing the body tracking data "routing" to multiple avatars ("skeleton" and "particles"), yesterday, I wondered if that could also be useful for other things.

Well, there was an obvious one: when the player is using a portal, I can have another avatar instance placed on the "destination" platform, so that the part of player avatar which is going through the portal can be properly seen.

Let's explain it a little better, starting from the beginning.

Right now, this is what happens when you enter a portal: parts of your avatar go behind the portal mesh which is showing the texture rendered by the "portal camera" as discussed in the past:

I didn't mention this until now, but honestly it's something that has been bothering me a little since I first implemented the most basic form of portal, during the PoC prototyping days.

Now, if I put a clone of the player avatar positioned on the destination platform, at the same platform-relative position, I should be able to see the parts which are now hidden by the portal. More precisely, I'm going to see the copy of those parts, coming from the clone.

Let's give it a try.

Ok, getting the basics working was relatively easy. I just had to create the copy, animate it with the same body tracking data, like I did yesterday, and apply the proper offset. Let's see where I'm at:

The video should show "the trick" pretty well. To test more effectively, I disabled the "particles avatar" and configured the "cloned" character as activated (meaning that they have the brighter material), which makes it obvious what mesh you're actually seeing when throwing your hands into the portal.

While I'm sure one could fake this in terms of rendering, doing some work on the shaders involved, there are advantages to the naïve approach I'm using. For example, is easy to have a slightly different mesh for the "other side" of the portal, or to make it interactive.

This needs some thinking in terms of gameplay, but I'm not far from being able to punch an enemy through a portal, for example.

Anyway, I'm not done yet. I need to

• make the cloned character visible only when it's behind the portal

• set the cloned character layer so that it works properly with the zone crossing, when opening a bridge portal

• set the cloned character to the same state of the original (activated or not)

• re-enable the particles avatar too

All done! Here's me putting my hands into two portals, a "normal" and a "bridge" one.

Of course, the difference is a bit less obvious now that I'm not using the "activated" material for the avatar on the other side of the portal. Still, the particles are only on the primary avatar, and the health/mana bar too. That's why the moment one gets teleported, the particles appear back (instantly) on the arms - that's when the cloned avatar disappear, and you're only seeing the primary one.

While testing, I remembered that didn't bother fixing the health/mana bar that I poorly attached to the player on Wednesday.

I also realized that I have an opportunity to link the smart particles to the portal management in a more sensible way (in terms of design and game world consistency).

But... I'm going to save this discussion for next time!

In the conclusion of my previous post, I wrote that I felt the need to do a bit of clean-up before proceeding with the smart particles.

So for today I'm just going to do some minor restructuring of the recent code, and of the Unity scene.

I did many minor improvements here and there, nothing worth discussing at length.

I also removed some old scripts which weren't useful anymore, and removed a few test nodes from the Unity scene.

I feel like it's always worth doing a bit of refactoring after getting something working, resisting the temptation to just move forward. The project stays more manageable and you exploit the fact that the reasoning behind it it's still fresh: you take the opportunity to make the code easier to understand for when "future you" is going to go back to make some changes, having forgotten lots of details in the meantime.

Is everything perfect after one afternoon of tidying up? Not by far. There's still a lot to do, especially about the portal management, but you gotta start somewhere.

2024-10-01 - Portal subsystem refactoring

Yesterday I decided to skip refactor and clean-up for things related to the portal system. I noticed things that weren't in the right place, but the changes needed didn't feel "minor", so they called for a dedicated day. Which is today.

While working with the enemies and the avatar, I have structured the code into a nice set of elements which deal with different responsibilities of an entity.

I have the concepts of state provider and state presenter, and around that I also built the test behaviours which let me test the visualization via inspector. Additionally, I have separated the "debug visualization" operations.

For example, in the case of an enemy, I have

• In the state management layer

  • `EnemyStateManager` and the related data types (`Enemy.Data`, `Enemy.EState`...)

• In the presentation layer

  • `EnemyBhv` handling the presentation of an enemy, using other sub-components as needed

  • `DebugEnemyBhv` handling the presentation of an enemy in debug mode (e.g. showing the health bars)

  • `TestEnemyBhv` allowing the test of the presentation via inspector

I did the portal subsystem before getting to this architectural organization, so it's a bit messier than this. Refactoring time!

I moved some transient data from the `PlayerStateManager` to the `PortalStateManager`, which was long overdue.

After making `PortalStateManager` self-sufficient in terms of the information needed to drive the portal presentation behaviour, I updated `PortalBhv` so that is a state presenter for the portal entity. Along the way, I removed a `PortalManagerBhv` which had no reason to exist anymore (I didn't write it, but yesterday, doing a similar process, I got rid of `PlayerManagerBhv`).

Abstractions help in keeping things manageable when they make sense, but sometimes one introduces some unnecessary ones, that provide no real semantic value. So, good riddance!

Tomorrow I'll continue the refactoring work dealing specifically with the portal surface rendering.

2024-10-02 - From PortalRenderer to CameraManager

Something that has always bothered me a little is that in the `PortalRendererBhv` I'm doing things which are not only related to portals. What am I doing that doesn't fit? Well, I'm changing the cameras settings, both in terms of culling layers and of rendering background.

I did it there because `PortalRendererBhv` was where I took care of positioning the "portal camera", so it was quick to do there other "more global" things too. Quick, but dirty.

Today I'd like to revisit `PortalBhv` and `PortalRendererBhv` and shuffle things a bit.

Ok, it wasn't easy, but I did a step in the right direction.

Unfortunately I had forgotten many details about this part of the code (which probably is one of the most complicated so far), so it took me a while to figure out what to do.

Until now, `PortalRendererBhv` was operated by `PortalBhv`, which had a direct reference to it.

`PortalBhv` passed the portal surface `MeshRenderer` to `PortalRendererBhv`, which took care of updating the material (setting the render textures and changing the proximity level parameter, for example).

Then, `PortalRendererBhv` also took care of managing the cameras, and not only the "portal cameras" that update the render textures, but also the main cameras (and the skyboxes).

In today's refactoring, I changed `PortalRendererBhv` to `CamerasManagerBhv`, now working as an independent subsystem. The part which worked on the portal surface `MeshRenderer`, updating its material, got moved into the main `PortalBhv`, where it belongs, as it's working on a child component of the portal node managed by `PortalBhv`.

`PortalBhv`, during its initialization, queries `CamerasManagerBhv` to get references to the render textures that it must assign to its surface `Material`.

There's a definite architectural improvement, because `CamerasManagerBhv` doesn't directly modify the portal, but just the cameras. `PortalBhv` now handles the `Material` and remains in charge of configuring the transforms which indicate entrance and exit of the portal, which `CamerasManagerBhv` uses to update the portal cameras poses (and, in turn, the render textures which get displayed on the portal surface).

Part of me wants to generalize all this so that the "portal cameras" are instantiated and moved directly by `PortalBhv`, but that would be worth it only if many portals could be opened at the same time, each with its cameras. I'm designing the game avoiding that possibility (which would definitely kill the framerate), so I'm going to contain myself and don't do it. At least for now!

Talking about moving portal cameras, I think I know what I'm going to do tomorrow...

2024-10-03 - Portal camera movement

Last week, on 2024-09-25, I wrote:

You might notice a problem, which I already spotted a long time ago but haven't faced yet. The "content" shown in the "distant" bridge portal is wrong. It doesn't feel like the portal is coming forward, but like it is a strange window into another space, which feels right only when completely opened. The effect is even worse in the headset, because of stereoscopy: you see the "close" mirrored avatar, but confined in the "far" hexagon, and your brain and your eyes complain.

Considering that yesterday I "refreshed" my understanding of the portal camera management, it should be a good moment to fix this.

Ok, I had to take pen and paper for a minute, but I did what I wanted.

The difference is subtle for the normal portals, but quite meaningful for the "bridge" portals which stay opened at a distance.

Let's try and see what I changed with a diagram, showing "before and after" the fix...

The yellow camera is attached to the player. The other is the "portal camera", which draws to the render texture which is then displayed on the portal surface. The portal starts appearing from the destination platform, and while scaling up to full size, it comes forward to the player platform. In the case of bridge portals, there's no scaling: the "distant" portal is always visible at full scale, and only comes forward.

The positioning of the portal camera is calculated on the basis of the player camera and of the current portal. I'm simplifying a bit, because the real setup involves more cameras, to take care of stereoscopy: right eye and left eye see two different textures rendered on the portal surface.

Anyway, in the upper part of the diagram (before the fix), the portal camera was always in the "destination" position, even when the portal was distant, which caused the problem described at the beginning, because you saw something that was supposed to be far as if it was "near" (cropped in the portal area).

Now, instead, I animated the portal camera so that, at the beginning of the animation, it matches the position of the player camera, and at the end, it's in the correct position to show the viewpoint as if the player were on the destination platform. Basically, the camera moves together with the portal, but in the opposite direction.

How does the result look? Let's see it with the portal opening and closing speed slowed down by a factor of ten.

And here's a short in-editor capture that shows what happens behind the scenes:

I added a red cube to make it easier to understand. In the bottom right corner, there's the portal camera render preview. In the scene view, you can see how the camera node moves together with the player, and then animates to the destination. When the player crosses the portal and gets teleported, the player camera matches what was shown by the portal camera and the magic trick is complete.

I like the resulting effect, but a little problem showed up: with this new camera behaviour, when there's a bridge portal that can be crossed both sides, the "cover" for the other sides is shown in the portal. Something that I had not anticipated, but that is completely correct considering how they are currently implemented and positioned. Here's a short clip showing the problem:

It doesn't show in the clips, but the new effect doesn't really work well with the portal scaling of the "non-bridge" portals. So I decided to disable it, at least for now. I'm going to do a different "appearance" animation. Between this and the need to fix the "covers", I guess I'll be pretty busy tomorrow.

2024-10-04 - Refactoring portal covers

Ok then, time to refactor the portal covers and fix the issue shown yesterday.

As first step, I'm going to rename a few things currently named "bridge portal" to "portal cover", because that's what it currently is: the bridge portal themselves are handled like all the other portals, with a couple variations handled in place for "bridge" and "mirror" portals.

So the red/green elements that show where a bridge portal is, and disappear when they're ready to open, are basically "covers" in front of the actual portal.

Well, that was a lot of renaming, but at least now the concept of bridge portal (a portal connecting two zones of the game world) and that of portal cover (an element visualizing if a bridge portal is available and the target zone is loaded) are well separated.

And now, the real work begins...

I started by aligning the "portal cover" with the other entities I recently worked on, so I restructured `PortalCoverBhv` as a state presenter and added a `TestPortalCover` behaviour that allows me to test via inspector the different states and the animations of the component.

This was also a good way to refresh my memory about how this thing worked and how it interacted with the gameplay simulation logic.

It was also a good opportunity to change the obnoxious red/green colors that I had used as test color to indicate the locked/unlocked state. Still temporary colors, but maybe they won't ruin all my screenshots and videos anymore.

About the actual modification, I think the easiest fix is setting the portal cover to the same state that I set when the portal is opened (so, when the cover becomes invisible). In theory, I should only need to set this state on the `PortalCoverManager` that handles "the other side" of the bridge portal... let's try!

Well, it's not pretty, but did the job:

            if (portalDstLoc.HasValue 
                    && portalDstLoc.Value.m_zoneId != playerLoc.m_zoneId) {

                PortalCoverStateManager rOtherSidePortalCover =
                    getWorldStateManager()
                    .getPlatformStateManager(portalDstLoc.Value)
                    .getPortalCoverStateManager();                
                if (rOtherSidePortalCover != null) {
                    rOtherSidePortalCover.setLocked(false);
                    rOtherSidePortalCover.setLoadingState(true, true);
                }

            }

So basically "if there's currently a portal bringing to a platform in a zone which is not the player zone, try to fetch the `PortalCoverStateManager` of the destination platform. If you find one, set it in a way that makes the cover disappear".

The `setLoadingState()` call is awful. I don't remember why I did it that way and not using two enumerations and not two booleans, like I usually do to keep things more readable. Well, that's a change for another day...

End of week video capture!

Ok then, thanks for reading yet another entry of the Particular Reality DevLog.

I managed to post each week for three consecutive weeks, but now I'm afraid I'm going to take a break for a week or two, and take care of a few things on my backlog.

As soon as I can, I'll be back at work, and I think I'm going to finally start development of the smart particles subsystem mentioned in week 25.

Let's continue from where we stopped. I need to

• detect if the player is touching/approaching the "mirrored" skeleton

• implement the special "mirror portal" logic

I would also like to do some kind of VFX on the portal surface, to make it react to the player's body, but that's not a priority at this stage.

I think the most natural thing to do is reaching into the mirror with the hands, but somebody could try with the head too. I mean, I guess they could also use the elbows.

I'd like to have an indication of the minimum current distance between body and mirror, so that I can modulate some effect to ease into the "threshold" point where the transition happens.

Basically, the problem reduces to calculating the distance between a point (a body part) and a plane (the mirror). Once I can do that, doing it for multiple elements and considering the minimum is trivial.

I did a quick test considering head and hands, and showing the distance values in the debug panel.

The Unity API offers a `Plane` class with a `GetDistanceToPoint` method, so I just had to initialize a `Plane` with the mirror position/normal, and fetch the head/hands position from the body input data.

For example, that's the code for the head distance:

Plane p = new Plane(m_vMirrorNormal, m_rMirrorTr.position);
Vector3 vHeadPos = BodyInputManagerBhv.i().getHeadPose().position;
float fHeadDistance = p.GetDistanceToPoint(vHeadPos);

Of course, I could do better, considering the actual meshes geometries, but that sounds like something for a future polishing phase, and after having the final 3D model so that I can fine-tune the threshold distances.

2024-09-24 - Mirror proximity indicator

I'm going to bring the test done yesterday to a state which I can actually use in the game logic.

I'm going to define two values which I will be able to easily tune:

• `farMirrorActivation`

• `nearMirrorActivation`

I will compare the minimum distance of those measured from body to mirror with this range, and derive a 0-1 value from there.

I will then use this value to modulate a "pre-transition" effect, and finally, when the value reaches 1, I will activate the "mirror portal" and teleport the player to the menuZone.

Calculating the 0-1 value was trivial, and I also did some restructuring so that the processing is done in the "gameplay logic" part of the code.

Yesterday, for testing purpose, I cheated a bit and did it accessing the body tracking directly and not from the "state management" layer.

That took some time, as I also ran into a bug and into something I had only partially implemented and had to be completed before usage. So, that's it for today!

2024-09-25 - Mirrored character placement

I have a mirror proximity indicator which I can use to activate the teleport to the menu zone, and I have the "mirrored" character.

But I don't have the "mirror portal" yet.

To get started, I'd like to use the bridge portal system I already have implemented, and just change some details about how the portal opens or activates the teleportation of the player.

If I manually open the bridge portal to the limbo zone, of course, the mirrored character becomes covered by the portal: to focus on the "mirroring" I temporarily placed it on the limbo platform, together with the player avatar.

It's finally time to move it to the destination platform, in the menu zone, so that it can show up only inside the portal.

Ok, I added a bit of logic to fetch the menu platform placement and used it to adjust the position of the mirrored character.

I also did the layer setup that I usually do to properly cull objects depending on the camera which is drawing them (the "normal" camera, or the camera for the portal render texture).

I also visualized the "proximity" index by setting the screen fade opacity, just for testing.

You might notice a problem, which I already spotted a long time ago but haven't faced yet.

The "content" shown in the "distant" bridge portal is wrong. It doesn't feel like the portal is coming forward, but like it is a strange window into another space, which feels right only when completely opened. The effect is even worse in the headset, because of stereoscopy: you see the "close" mirrored avatar, but confined in the "far" hexagon, and your brain and your eyes complain.

In my next iteration on the portal rendering, I will get it fixed. I'm tempted to try and do it now, but I don't want to wander off the path.

...and the Meta Connect keynote is about to start, so that's it for today!

2024-09-26 - Proximity indicator VFX

Well, the Meta Connect news about the Orion AR glasses prototype were pretty cool. On the VR/gaming side, nothing spectacular, but some good little things. And I think the Quest 3S has potential to further expand the VR user-base, which of course is a good thing for the medium. And more potential players for _Particular Reality_, yay!

So, let's get back at work, or it will never ship!

We have the mirrored character properly positioned, we have an indicator of the player proximity to the "mirror portal", and we have an avatar ready to change visualization mode.

Two things are still missing:

• some kind of effect to bind to the proximity indicator: yesterday I used the opacity of the white quad for quick testing, but that's reserved for the mode transition when going from limbo to menu (and changing avatar)

• some custom logic for the special "mirror" portal behaviour: it must open without requiring the opening gesture, and the teleport should happen depending on the proximity indicator and not the portal crossing

Let's proceed in order.

So, what kind of effect do I want to activate depending on the proximity indicator?
I think that it should be something on the portal itself. The player should understand that by reaching into it something might happen.

I don't feel very creative in terms of VFX today, so I went for something super-simple that I will definitely replace at some point.

I passed in a `float` value to the shader that draws the portal contents, and used the value to fade between greyscale and normal color.

Simple enough to take the chance and throw in a fragment shader snippet:

float lum = dot(col.rgb, float3(0.2126, 0.7152, 0.0722));
float3 gray = float3(lum, lum, lum);
float3 c = lerp(gray, col.rgb, _ActivationLevel);
col.rgb = c;

I calculate the luminance of the color, and use it to define an appropriate gray value.

Then, I linearly interpolate between the "normal" color and the grey depending on the `_ActivationLevel` passed from outside.

If you're wondering about how the luminance calculation works, and what does the dot product have to do with it, the answer is simple enough to include here.

The luminance is the "perceived brightness" of a color, and can be calculated by modulating the values of its RGB channels for some constants.

For a "linear" RGB color `c`, the luminance `lum` can be calculated like this:

lum = (0.2126 * c.r) + (0.7152 * c.g) + (0.0722 * c.b)

If you remember the dot product formula, you already understood why that works.

If you don't, well, for two 3D vectors `a` and `b`:

dot(a, b) == (a.x * b.x) + (a.y * b.y) + (a.z * b.z)

So, it's exactly the computation we need for the luminance, with `a = (0.2126, 0.7152, 0.0722)` and `b = c.rgb`.

2024-09-27 - Mirror portal activation

It's finally time to face the core challenge, which is implementing custom logic for the "mirror portal".

I already have a (bridge) portal configured to bring the player to the menuZone.

Right now, all the portals work like this:

• when the player is in a suitable position of the platform and performs the "opening" gesture, they appear/open

• when they are crossed, the player is teleported to the destination location

So, the player positioning check, and the teleportation to the destination, also apply to the special "mirror" portal.

Instead, I need to provide alternatives for:

• opening the portal (it should open automatically)

• activating the player teleportation (it should activate when the proximity indicator reaches value 1)

As first step, I'm going to be shameless and hardcode the special logic behind a couple of conditionals. I have a very simple way to check if the portal available on the current platform is the "mirror" portal, which is just checking if the source or target zone of the portal is the limboZone.

I added the logic and it worked but... not everything went smoothly as expected! When I get to the destination platform, I find myself in a state which is detected as crossing back from the menu to the limbo with the "normal" bridge portal logic. I didn't think about it, but it makes sense if I consider how the normal "portal crossing" detection works.

Considering that I'm changing the way one exits the menu, I can remove that bridge portal from the menuZone definition and just avoid the problem.

Cool, it works! Time to test a Quest build in standalone mode and see if there are any problems that don't show up testing in-editor via Meta Link.

Guess what? It didn't work well! I think it's something related to the calibration offset... let's try to figure it out.

Ok, after a bit of debugging I managed to get it working consistently. There were a couple of things about how the reflection was calculated, and a world/local position I got wrong but worked anyway if there was no calibration offset, which explains what happened.

Let's see a capture on device - I disabled the passthrough for the capture, because the room is a mess, but that works too.

And another week is gone! I'm quite happy about the progress, but the code quality isn't great because I rushed things a bit. I wanted to see it done before the week ended, and I did it, but I took some shortcuts.

Next week I'm going to do a bit of refactoring and clean-up, before proceeding with the smart particles.

In week 25, I did a lot of thinking and wrote down many ideas. Now it's time to work on making those ideas part of the game.

Before starting to actually write code, let's prepare a list of the things I need to do, at a surface level.

So, I need to:

• improve significantly the player avatar management

• make the special "mirror portal" between limbo and menu zone

• make the "white flash" transition between limbo and menu zone

• implement the dynamic "hint system" with particles pushing the player towards a specific spot, and showing what gestures to do

• implement the navigation between contexts and the application flow shown by the FSM diagram at the end of the article

That's a lot of work, but as long as I manage to reasonably divide it into tasks to be faced one at a time, and put together at the end, it should be fine.

I'm going to start from the avatar.

I think that I reached a good synthesis of what I need to do about the avatar in a table I showed last week:

Basically, the `Context` and `World` tell when a certain avatar/particles setup operates and that will come into play later.

Now, we have to work on the `Avatar` and `Particles` so that they can be in the different required configurations, each with its behaviour.

Even if they will ultimately work together, we should be able to work on the two subsystems separately.

I feel like we need a name to distinguish "these" particles from the others. Let's go with "smart particles", as they will actively do things and not just be a cosmetic or passive element of the game.

So, two separate subsystems, that will work together, because normally the smart particles will follow the avatar.

Let's start from the avatar management, that I already have implemented in a basic form, and that I will now refactor. The table indicates three possible avatar modes, so let's define an identifier for each mode, to keep things short and sweet:

• Semi-transparent hands: `limboHands`

• Skeleton (inactive): `inactiveSkeleton`

• Skeleton (active): `activeSkeleton`

According to what I defined in week 25, the player can't go directly from gameplay to limbo, but must always go through the menu zone. This means that we don't need to define a direct transition between `limboHands` and `activeSkeleton`: the avatar must always go through `inactiveSkeleton`, like this:

The kind of handling I need to implement to take care of these state changes is easier than most things we've been dealing in the past weeks.

Why? I don't need to think about state persistence, possibility to rewind gameplay etc.

I'm basically dealing with a state element which is not part of the gameplay state, but of the application state. Something which is intrinsically transient and won't be stored in a savegame.

This doesn't mean that I will be totally sloppy, and I'm still going to define a simple FSM which supports non-instant transitions. That will come useful when I need to orchestrate the state change of different elements when implementing the global application flow.

Let's complete the day with a really NOT interesting FSM diagram which makes the transition states explicit and asymmetric, so that by reading the FSM state is immediately obvious not only if a transition is happening, but also which way it's going.

This won't force me to be fancy in the presentation layer, but if I decide to be, the logic will be there already.

2024-09-17 - Avatar FSM implementation

Time to do some actual implementation work.

As first step, I renamed what I've been calling `PlayerAvatar` until today to `PlayerBody`, as it was pretty much a description of the head/hands and bones of the player body, as used not only to drive the avatar but in the motion clip capture and playback.

So it was not very appropriate anyway, and I can now use that same name in the right context: the avatar management.

I went and implemented the FSM I designed yesterday. As I'm doing recently, I also implemented a `TestPlayerAvatarBhv` to test the state changes via inspector.

There's a bit of boilerplate in the code, which bothers me a little, but I feel like the system is becoming quite robust. Everything worked correctly on the first try.

While implementing the FSM, I noticed that there was quite some repetition: basically, the states are of two kinds, the three "core" which represent the normal visualization of a certain mode, and the "transition" states which will be presented with some kind of animation between the modes.

The execution exits the "core" states when the right input signal comes, while the "transition" states automatically send to another state after a specific delay.

We're talking about very quick animations, and considering that they should be activated when navigating through contexts as previously explained, they shouldn't need to be interruptible or reversible, which keeps things simple.

Let's revisit the diagram - I recently learned how to apply colors to states in Mermaid diagrams, and hey, I'm going to make use of that.

I labelled the transitions so that they show the input signal needed to change state (on the arrows coming out of the cyan states, the "core" ones), or the constant representing the duration of the transition (on the arrows coming out of the magenta states).

In a resource constrained environment, I would implement this FSM logic differently, in a more compact way that just uses two `currCoreState`and `nextCoreState` variables, and a flag and a timestamp to handle the transition logic.

Instead, I'm sticking to the approach used in the other FSM implementation shown so far.

This doesn't mean that I can't avoid repetition in the code: I just need to define a couple of helper methods to generate the two kinds of state function I need, but with different parameters:

private static Func<IPlayerAvatarFSMView, EState?> getBaseStateFunc(
    Dictionary<EInput, EState> rTransitionsMap
) {
    return (IPlayerAvatarFSMView rExec) => {
        var rData = rExec.getExecData().getData();
        if (rExec.isInputAvailable()) {
            FSMInput<EInput> input = rExec.fetchInput();
            EState eState;
            if (rTransitionsMap.TryGetValue(input.m_id, out eState)) {
                return eState;
            } 
        }
        return null;
    };
}

private static Func<IPlayerAvatarFSMView, EState?> getAnimStateFunc(
        float fDurationSecs,
        EState eDstState
) {
    return (IPlayerAvatarFSMView rExec) => {
        var rData = rExec.getExecData().getData();
        float fProgress =
            rExec.getExecData().getCurrStateElapsedTime() /
            fDurationSecs;
        rData.m_fAnimProgress01 = Mathf.Clamp01(fProgress);
        if (rData.m_fAnimProgress01 == 1f) {
            return eDstState;
        }
        return null;
    };
}

Then, I can simply define the state functions for the whole FSM in a very compact manner.

It would be cool (and not that hard) to automatically generate this definition from the Mermaid diagram text... but not this time.

private static 
    Dictionary<EState, Func<IPlayerAvatarFSMView, EState?>> rStates =
    new Dictionary<EState, Func<IPlayerAvatarFSMView, EState?>>() {
        {
            EState.showing_limboHands,
            getBaseStateFunc(
                new Dictionary<EInput, EState>() {
                    {
                        EInput.setSkeleton, 
                        EState.limboHands_to_inactiveSkeleton 
                    }
                }
            )
        },
        {
            EState.limboHands_to_inactiveSkeleton,
            getAnimStateFunc(
                fSETSKELETON_DURATION_SECS, 
                EState.showing_inactiveSkeleton
            )
        },
        {
            EState.showing_inactiveSkeleton,
            getBaseStateFunc(
                new Dictionary<EInput, EState>() {
                    { 
                        EInput.activateSkeleton, 
                        EState.activating_skeleton 
                    },
                    { 
                        EInput.setLimboHands, 
                        EState.inactiveSkeleton_to_limboHands 
                    }
                }
            )
        },               
        {
            EState.activating_skeleton,
            getAnimStateFunc(
                fACTIVATESKELETON_DURATION_SECS, 
                EState.showing_activeSkeleton
            )
        },
        {
            EState.showing_activeSkeleton,
            getBaseStateFunc(
                new Dictionary<EInput, EState>() {
                    { 
                        EInput.deactivateSkeleton,
                        EState.deactivating_skeleton 
                    }
                }
            )
        },
        {
            EState.deactivating_skeleton,
            getAnimStateFunc(
                fDEACTIVATESKELETON_DURATION_SECS,
                EState.showing_inactiveSkeleton
            )
        },
        {
            EState.inactiveSkeleton_to_limboHands,
            getAnimStateFunc(
                fSETLIMBOHANDS_DURATION_SECS,
                EState.showing_limboHands
            )
        }
    };

2024-09-18 - Avatar presentation layer

Now that I have the avatar mode logic in place, and a way to activate it through the inspector, it's time to work on the presentation layer.

For the "limbo hands" I'm just going to use the default hands offered by the SDK.

For the "skeleton", I'm going to use the robot character from the Meta samples I started with, changing something on the material to show the activated/non activated modes.

I did a few changes to the scene and added two simple scripts controlling the two elements which come into play: `LimboHandsBhv` and `SkeletonAvatarBhv`, each connected to the relevant nodes.

Then, I implemented the "state presenter" behaviour for the player avatar, which reads from its state manager and configures the nodes which take care of the presentation.

The visibility change of the hands and of the skeleton is instant for now, even if I pass in a 0-1 value to indicate the progress when in a transition state. In the case of the skeleton activation/deactivation, I use such value for a simple color interpolation.

At some point, when I'm going to have the actual, custom 3D model for the player avatar, I will do something fancier.

Everything is quite simple and tidy:

• `LimboHandsBhv` controls the visibility of the semi transparent hands

• `SkeletonAvatarBhv` controls the visibility and "activation" of the skeleton body

• `PlayerAvatarBhv` reads the avatar state data and uses it to properly configure `LimboHandsBhv` and `SkeletonAvatarBhv`

Here's a short clip of me checking that everything works properly using the `TestPlayerAvatarBhv`.

Notice the small jump in the hand position when switching between hands and skeleton. You might remember about the mismatch that got introduced when updating the `com.meta.movement` package to version `5.0.1`. I checked on GitHub and the current release is `6.0.0`. I wonder if they fixed it? It might be worth checking, hoping there are no big breaking changes (major version bump!).

Anyway, it's going to take a while to have the avatar mode switching normally working in-game as specified by the design: we need to build some other pieces of the puzzle first.

2024-09-19 - Mirrored body

Let's focus on one small part of the design work done in week 25:

With pretty much nothing else to do, the player will have to walk into the portal, or at least reach into it with their hands. When he "touches" the reflected avatar, in a white flash, I will teleport the player to the "menu" zone, and they will find themselves embodying the avatar which was "on the other side" one moment before that.

So, how do I implement this? Let's see what do I have, and what's missing.

I have

• a working portal system which allows me to "see" into the destination zone

• a way to drive characters from body tracking data (live, or saved)

I don't have

• logic to detect the conditions to activate this special teleport (the player is not walking into a portal but is just "touching" it)

• the "white flash"

• a way to "mirror" the body tracking data

I can place a copy of the skeleton avatar on the destination platform (that gets shown in the portal on the limbo platform), but if I don't flip the avatar properly, the illusion won't work.

Luckily, I remember the mirroring being part of one of the samples. Time to revisit them and see if they can save me some work.

I checked the sample I remembered about, which is in the `MovementRetargeting` scene of the `com.meta.movement`package.

The implementation involves:

• a `MirrorTransformation` script, which lets you define mirror normal and plane offset, and links to a `_transformToMirror` which acts as parent transform of the "mirrored" objects

• a `LateMirrorerdObject` script to be attached to a "mirrored" object and linked to the "original" one, which lets you define a list of `MirroredTransformPair`, where each pair links an "original transform" (of the object to mirror) to a "mirrored transform" (of the mirrored object), and copies the original transform to the mirrored transform

In the case of the player character, in the scene there's another instance of the character model used for the player, under the node set as `_transformToMirror`, and on its `Skeleton` node there's the `LateMirroredObject` which links it to the player character model.

I should be able to replicate this setup pretty easily. The only problem is that in the `limboHands` state I wouldn't have an "original" body to feed to `LateMirroredObject`, because the skeleton avatar is not actively tracking in that state. Maybe I can change it so that is active but not rendered? Let's try.

Ok, I got it working!

I did what I anticipated, with the only detail that "active but not rendered" didn't mean turning off the `SkinnedMeshRenderer`, which needs to be active to properly update the bones. What I had to do, instead, was keeping it active but drawing it with the "null shader" shown on 2024-02-14.

I didn't have to touch anything except the implementation of a `setShown()` method I had defined in `SkeletonAvatarBhv`.

There's still a few things to get done before considering this "mirroring" feature done, like placing the "mirror" transform relatively to the limbo platform (for now I just placed it at zero, which works only if there's no play area offset manually).

I'll get back to it soon.

2024-09-20 - White flash transition

Let's proceed in building the elements we need. Today I'm going to tackle the effect to be used when transitioning from limbo hands (in the limbo zone) to the skeleton avatar (in the menu zone).

I'll follow my usual implementation principle, which is doing things in a basic way but with a sensible interface, and maybe replace them in the future with something better.

I want to be able to tune the transition timing and have a "safe" interval where I know that the player can't see anything, allowing me to change things around them without breaking the immersion.

The most basic thing that comes to mind is a full screen white flash that I can control via code setting the transparency level.

So, at the end, I'm going to do something like:

• when ready, fade to fully opaque white in 0.2 sec

• when fully opaque, set the avatar mode / teleport to the menu zone etc

• finally, fade to fully transparent in 0.1 sec

So, how to implement the white flash?

The most crude way, which I've used sometimes in the past adapting a classic non-VR approach, is sticking a quad drawn with a shader supporting transparency in front of the camera, and set its color.

After a brief search, I noticed that there's an `OVRScreenFade` script offered by the SDK that basically does that, creating the geometry via code, which is nice.

Let's try to use it...

I want control over the transparency level and timing of the fade, so I didn't use the `FadeIn` / `FadeOut` methods offered by `OVRScreenFade.cs`.

Actually, I made a super-thin wrapper over it and I'm going to use that whenever I need the functionality.

using UnityEngine;
using BinaryCharm.Behaviours;

namespace BinaryCharm.ParticularReality.Presentation {
    public class ScreenFadeManagerBhv : ManagerBhv<ScreenFadeManagerBhv> {

        private void Awake() {
            registerInstance(this);
        }

        public void setOpacityLevel(float f01) {
            OVRScreenFade.instance.SetExplicitFade(f01);
        }

        public void setFadeColor(Color c) {
            OVRScreenFade.instance.fadeColor = c;
        }
    }
}

In the `PlayerAvatarBhv` script, which "orchestrates" the other components related to the avatar presentation, I called `setOpacityLevel` remapping the progress level of the transition states `limboHands_to_inactiveSkeleton` and `inactiveSkeleton_to_limboHands`.

It sounds like a good moment to throw in a cool little trick: to remap a 0-1 value to a value which instead goes up and down (so that we can have maximum opacity in the middle, and have a fade out/fade in) I did:

// fAnim01 is the progress level of the transition
fFadeOpacity = 1f - Mathf.Abs(2f * fAnim01 - 1f);
ScreenFadeManagerBhv.i().setOpacityLevel(fFadeOpacity);

An equivalent formulation, using a conditional, would have been:

if (fAnim01 < 0.5f) {
    fFadeOpacity = fAnim01 * 2f;
} else {
    fFadeOpacity = (1f - fAnim01) * 2f;
}

...that could be written in a more compact way using the ternary operator:

fFadeOpacity = (fAnim01 < 0.5f ? fAnim01 : 1f - fAnim01) * 2f;

Using the absolute value is a bit more opaque, but feels cleaner and avoids branching.

Not that would matter here, but it's the kind of thing which is nice to know and can be much more important in the context of shader code. Still, when I do this kind of thing, I try to leave as a comment the equivalent, explicit formulation.

Another week of (part-time) work on Particular Reality is done. Let's complete this DevLog entry with a short clip:

At some point I will probably add an easing function to the value, but even in the current state, I feel it's adequate.

Next week I'll continue the work, taking care of the "touching mirror detection" and, hopefully, of the "mirror portal" teleportation, so that we can see it all come together.

I have a busy week, but I'm not going to stop completely!

I feel it's the right week to do a little design.

In week 24, I implemented enemy attacks which can lower the player health. What happens when such health goes to zero? It's time to revisit and evolve the global application flow..

How can I stay faithful to my "total immersion" design principle, and still have a familiar "game" flow with save/load, "runs", pause, and maybe rewind?

I've been thinking about these things for quite some time, and it's time to write about them and flesh out the details.

Remember week 7? I defined two special zones, "limbo" and "menu".

The "limbo" zone should be the zone which acts as a passage from the outside world (or the VR headset "home" environment) to the game world, while the "menu" should let the player start a new game or resume playing a previous game.

In these two "special" zones, it should be acceptable to have some kind of UI elements or messages, but I'd like to keep them minimal anyway.

I need to put myself into the shoes of somebody that has never played or seen the game, and think about what to show them when they start the game.

Loading the "limbo" should be almost instantaneous, and the hexagonal platform should appear in mixed reality. The *Binary Charm* and _Particular Reality_ logos could also float somewhere. This is where I might need to show any disclaimer/warnings about potential epilepsy triggering etc.

Additionally, I will need to make sure that the platform is inside the "guardian" area, and let the user adjust it if they want. Any other configurable options, if any, will go here too.

In the _limbo_, there should basically be no avatar, but only the default semi-transparent hands, just to make the player aware that the hand tracking is working.

Then, I need to push the player roughly near one edge of the hexagonal platform, and facing the opposite edge. This way, the portal to cross, which is going to "split" the platform in two halves when it appears, will open in front of the player, well visible.

Having some kind of arrow stamped on the platform would be effective, but too crude for my taste. What I'm thinking is having a dynamic hint system, based on floating particles, that goes into the player field of view and moves trying to push him into a certain spot.

When the player reaches the required position, a portal splitting the hexagonal platform in two halves will appear. It's going to be like a mirror showing our reflection from another reality: a Particular Reality.

Our reflection should feature what I'm going to call a "skeleton avatar", a minimalistic, stylized upper body that will always track the player body movements after they enter the game world.

With pretty much nothing else to do, the player will have to walk into the portal, or at least reach into it with their hands. When he "touches" the reflected avatar, in a white flash, I will teleport the player to the "menu" zone, and they will find themselves embodying the avatar which was "on the other side" one moment before that.

Obviously, this should resonate with the player, thanks to Alice in Wonderland (and, consequently, Matrix), without forgetting the classic moment from Prince of Persia.

Then what?

2024-08-27 - Teaching locomotion

Ok, yesterday I got to the point where the player goes from the limbo to the menu zone by crossing the portal/mirror, and starts embodying the "skeleton avatar".

In the menu zone, the player should be able to start playing a new game, or resume a previously saved game. Entering this "playing" mode should be made obvious by having a second level of embodiment, where the "skeleton avatar" binds itself to some kind of "flowing energy" (rendered via particle effects).

But first, the player should be taught at least one fundamental thing: how to move between platforms.

For the second time (forced action repetition is the basic way to teach the rules of the game world), the player, standing on the second half of the first platform of the "menu zone", after entering from the limbo, will be drawn to the zone where they can open a portal to the adjacent platform.

When in position, the floating particles will morph to the gesture needed to open the portal, prompting the player to do that same gesture.

When the "skeleton avatar" hands match the "particles hands", they will activate the portal to the adjacent platform.

As we know from one year of DevLog, the player should take a step forward while sticking to the same gesture.

We should teach this by having the "particles hand" go forward into the portal, pushing the player to follow in the same way.

After reaching the central platform, the player will need to reiterate the action, but changing direction. So we're slightly making things more complicated, trying to teach "to move to an adjacent platform, you must be in correct half of the current platform, facing towards the destination". The floating particles will suggest the actions to be done, as they already did on the previous platform.

At this point, the player will be on the "new game" platform, which will feature a "bridge portal" which brings to the start of the actual game.

Once again, the floating particles will still suggest what to do. When the player will cross this portal, marking the beginning of a playthrough, the particles which have "guided" the player until now will stick to them, somewhat "activating" the skeleton and marking the start of the gameplay session.

Considering the menu map (without the "continue game" platform), here's a visual representation of the three portal crossings we are going to guide the player through, starting from the bottom and reaching the "gameplay" on the top. The red arrows are movement on a platform (the "step back"/direction change), while the azure crayon shows the "jumps" done crossing an opened portal.

2024-08-28 - Navigating through contexts

Let's recap the flow we have described until now:

• the game starts in the mixed reality "limbo", the player doesn't embody an avatar

• the player moves to "menu" and embodies the "skeleton avatar", particles float around and give hints

• the player enters the "gameplay" stage, and the "skeleton avatar" activates, with particles sticking to it

In the "gameplay" stage, there's a sub context to handle, the one where the game is paused (and maybe gets rewind).

Let's split "gameplay" into "active gameplay" and "paused gameplay", and recap how I'm planning to make each context easy to tell from the others:

Basically, when the game gets paused, the particles should "detach" from the skeleton, standing still even if the player (still driving the "skeleton avatar") moves.

I'd like to have a single "pop context" interaction, that the player can perform to "go up" one level in the stack, like this:

It's like when pressing "esc" pauses the game, and then pressing it again brings you to the main menu. There, another "esc" keypress quits the game.

As there's no standard "esc" interaction for VR, and I'm trying to avoid relying on controllers and "system" gestures, the only problem is finding the right gesture/movement, teaching it to the players, and make them aware that the possibility to perform the interaction is available.

Not relying on the system gesture (which currently, on Quest, is a pinch held for a while) doesn't mean that we don't have to handle it, of course. Actually, by being an intrinsically "immersion breaking" interaction, it might provide me a good opportunity to show a panel teaching how to do an `exitContext()` interaction, if I can't think of anything better.

Anyway, I feel like it's time for one of my FSM diagrams. Let's try to describe the global application flow.

A few comments on the diagram, before calling it a day:

• The `in_limbo_shutdown` is not really necessary, but I'd like to have it, for symmetry: I'm going to implement a quick animation where you exit the Particular Reality and are briefly sent to the mixed reality you've already seen when launching the game. If you're in a hurry, you can always quit the game straight from the system menu. I'm not sure yet about giving or not the option to go back to the "limbo" without quitting. For now, I'm going to keep it simple and assume that "exiting" from the menu zone implies wanting to quit the game.

• Walking outside the hexagonal platform is another way to put pause the game, and solves problems like "what happens if the player has a large room and walks to an adjacent platform?".

• I haven't considered the "setup" logic yet. I briefly mentioned yesterday, and it's part of the global application flow, so I should formalize that too

Well, I'm going to continue tomorrow...

2024-08-29 - Unfinished business

While reviewing yesterday's work before proceeding, I spotted a flaw.

I wrote that while in the "active gameplay" context, a second way to pause would be to simply walk outside the platform, which would also prevent breaking the game world logic, which forces you to change platform via portals.

But what about walking outside when in the other contexts?

• limbo: it's fine, you're basically walking into your room in mixed reality, but to proceed to the game you must step onto the platform

• paused gameplay: it's fine too, you might walk around but to resume playing you are forced to go back to the platform

• menu: problems! problems! eeek!

If I want to keep some level of consistency, I think that "exiting context" in the menu zone (including stepping out of the platform) must definitely bring back to the limbo, which then is not just for starting and quitting the game, but can potentially be entered and abandoned again at any moment.

When stepping outside the platform while in the "menu" context, the "skeleton avatar" must be detached, the environment should go back to mixed reality, and the player should go back to have the semi-transparent tracked hands.

When going back inside the platform, the "mirror portal" will open again, allowing the player to re-embody the "skeleton avatar" and go back to the same platform they were before switching to the limbo.

If I'm not overlooking anything else, this slightly more complex flow should make sense, and it's now possible to quit the game from the startup scene too, because for additional consistency I would say that at this point walking outside the platform, when in the limbo, should be the action that quits the game.

Let's update yesterday's diagram, and also take the opportunity to add the "setup" part.

Ok, looks like a definite improvement!

I got fancy and assigned different colors to the states, depending on the context they're in, according to this mapping:

I added a preliminary check for the validity of the configuration, and a new `in_limbo_config` state to take care of fixing it when needed. Then, I added the possibility to go back to the limbo from the menu. Now, `exitContext()` should be able to work consistently, always mapped to the same interaction, which should be "do a gesture/movement OR step outside the platform".

Of course, to activate the signal by stepping outside the platform, the player will need to step back in first to activate it again (or at each frame outside the platform, we would pop a context, which would quit the game in a handful of frames).

2024-08-30 - Back and forth

I'd like to spend some time thinking about what should hopefully be the last "cloudy" parts about the high level application flow.

I mentioned the "continue game" platform a few times, but never stopped and described how I imagine it working.

And in the diagram, there are two ways to go back to the `in_menu` state:

• from `paused`, voluntarily performing an `exitContext()`

• from `playing`, in case of game over

Let's discuss them separately.

Going to the menu zone from the `paused` state is the easy one. I know how I like this kind of auto-saving system: when I go back to playing, I want to be able to resume the game seamlessly, exactly from the state it was when I had to stop playing.

There's all kinds of things that can force someone to interrupt a gameplay session (battery running out, unexpected call, dinner ready...), and I hate when a game doesn't allow to easily save and then restore later. Forcing the player to leave the application running, paused, is not acceptable to me.

So, I'm just going to autosave whenever one pauses the game. Done that, it doesn't matter what happens next: maybe they quit regularly passing through the limbo, or more abruptly terminating the process from the system menu. Maybe they leave the game running, even if paused, in the headset until the battery runs out.

Let's define some further details. When "exiting the context" of the paused game, we defined that the players find themselves in the menu zone, but where exactly?

I'd like to put them on the "continue game" platform, and I'd like that the "bridge portal" that one can open from that platform is nothing else that a way to step back into the "paused game" that they have just left.

Actually, I'd like the bridge portal to act like a 3D thumbnail of the saved game. When you open it by stepping on the right zone of the "continue game" platform, you can glimpse into the portal and you might recognize the detached particles of your avatar as you have left it when pausing the game. Crossing the portal brings you back to the same "paused" game you had left, and you can reattach your "skeleton avatar" to the particles and restore the execution.

It doesn't matter if you had just paused, in the same execution of the application, of if it's the game you were playing two weeks earlier, before going on vacation.

What happens if you have a saved game, and consequently the possibility to continue but instead you decide to go back to the "new game" portal to start from scratch?

That's a whole other can of worms, let's leave it closed for now, and assume that we handle a single slot: "new game" overwrites it.

What about the "game over" case, when the player's health goes to zero?

I feel it's early to take a final decision about this.

There's many possibilities. I'm going to write some that I might consider.

• A checkpoint system. When you lose all your health, you get sent back to the latest checkpoint, with full health. The latest checkpoint might be the current zone entrance.

• Forced rewind: you never really "lose", but are forced to rewind to a previous state where you have some health. Might be problematic (limits on the rewind duration), and I'm not even sure rewinding will exist

• Game over and that's it. You need to restart the "run" from the beginning, maybe keeping some element of progression from the previous attempts

I really don't know yet. But even about this, like for the save/load, I need to at least take a temporary decision which makes sense for now.

I think I'm going to go with the simples thing, which is the "do nothing" option. I will send the player back to the "new game" platform of the menu zone. They just lost, and can start a new game.

Let's update the diagram to include the newly discussed logic:

The non-coloured states indicate behind the scenes actions that are significant to make the logic work the way I described earlier.

I had to remove the "macro-state" gameplay, because there's no common flow going in nor out.

• if we are entering from the "new game" portal, we set up a new game and go into the `playing` state

• if we are entering from the "continue game" portal, we set up the loaded game but go into the `paused` state, not straight into `playing`

• if we are exiting because of the "game over", we are brought back on the "new game" platform

• if we are exiting voluntarily with an `exitContext()` from `paused`, we are brought back on the "continue game" platform

Additionally, I highlighted the `autosave` logic called when pausing the game.

Ok, that's about it! There are going to be details I have overlooked, but I think I have drafted quite a lot of stuff, which is going to keep me busy for many weeks of implementation.

I can't wait to get started, but well, I will have to, actually, because next week, and maybe the week after that, I will be busy with other tasks. Hopefully, once I get rid of those, I will manage to work for a few weeks without interruptions.

I enjoyed preparing the special post and clip montage last week, looking back at one year of development, but now it's time to get back at work.

Since week 20, I've been reintroducing the enemies into the game, following the same principle where I swap out a crude, throwaway implementation done for the PoC, for a new, robust and well engineered implementation.

At the end of week 23, I was faced with a problem: I do an "invisible" move of the game world elements when entering a bridge platform, so that the current zone is always roughly at the world origin. But this movement impacts the calculation implicitly done in the visual effects, causing a glitch in the particles update. At least, that's my assumption, but I'm reasonably sure that's the problem.

Let's see what options do I have, and pick one.

• ignore the issue: it's a visualization thing, so I could live with it and go back at it later, when working on the presentation

• change the gameplay to hide the issue: make the enemies appear only after the player enters the zone, so they would be hidden during the zone repositioning

• remove the zone repositioning: I did it thinking about the classic problem of float precision degrading with distance from zero, and to ease debugging in the editor. But it's not a "real" fix - ultimately, I'd like to use fixed point arithmetic and have deterministic calculations for both time and space

• handle the problem in the VFX: by passing as parameter the needed information (the velocity vector, or maybe the base transform of the zone) it should be possible to adjust the calculation so that it doesn't glitch

I think I'm going to try the last option. But why?

I don't want to ignore the issue, because this is not a glitch that would disappear just using prettier models and better visual effects down the road. I don't want to take gameplay decisions because of technical problems, so option two is also off the table. Option three would be great, but it would be a lot of work to implement fixed point arithmetic. And it would be wasted work if I end up ditching Unity (as I hope to do) at a later stage of development, or if they add it themselves (I read somewhere that it might get introduced to support deterministic physics in DOTS, or something like that).

Option four, instead, has these advantages:

• gaining more experience with the VFX graph would be useful anyway

• this kind of manual adjustment of the position could help me to also understand how to control the time delta of the simulation, something I wanted to do last week (I ended up "just pausing" the VFX).

Ok, enough writing and thinking, let's try to get it done.

Before trying to do it from scratch, I googled a bit and found out I'm not the only one looking to do this kind of thing. Actually, there's an example on the Unity forums that should show how to work around this limitation of the VFX graph. The "old" system, Shuriken, offered not only "local" or "world" simulation space, but also a third option, "custom", that let you specify a transform to use as basis for the particle system. Which is basically what I need to do with the "zone parent" node, which is the one I apply the offset to when entering a zone.

I tried the example and it should be doing what I want, but it's not straightforward to apply it to my enemy VFX, and it's getting late. I'm going to try it first thing tomorrow.

2024-08-13 - Playing (and failing) with the VFX graph

Let's play a bit with the VFX graph and see if I manage to apply the "custom simulation space" fix I found yesterday.

Unfortunately, it looks like it's "one of those days" where things just don't want to work.

I tried implementing the workaround in my enemy VFX, but it's not working properly.

After an afternoon of fiddling with nodes (which is something I hate), I'm not sure if I simply have some errors in my setup, or if what I'm trying to do is intrinsically wrong.

I think I'm almost there, but I'm at a point where to get it working with the workaround I have implemented, I should change the scene hierarchy so that the nodes with the `VisualEffect` component attached aren't descendants of the zone nodes, and I hate the idea.

While I managed to decouple the transform that should affect the particle simulation (the local movement of an enemy inside a zone) from the transform that should not affect it (the transform of the zone, that I adjust when crossing a portal from one zone to another), the current node hierarchy gets in the way.

Still, I feel like this is taking too much time and is not bringing me a "clean" fix but an ugly mess. Some test nodes are disconnected, but this is a summary image of my miserable day:

These two days haven't been completely wasted, because I definitely understood some new things about the VFX graph. But I think it's time to take a step back, because I can foresee lots of additional work to have each effect implementing my dirty workaround.

What step back? Well, I think I will fall back to "option three": disabling the zone repositioning. I explained that it wasn't a real fix anyway, so I'll get rid of it and implement the fixed point arithmetic someday (not in the prototype).

2024-08-14 - Disabling the zone repositioning

I started the day by tidying up a bit after the last two days of struggle.

I committed my "almost done" version and then disabled the additional calculations I introduced, practically going back to the previous version of the VFX.

Now, it's time for node-editor detox and finally doing some progress.

Disabling the zone repositioning took a few minutes and, of course, avoided the glitch.

I put the feature behind a flag, so if I want to re-enable I just need to change a constant.

That might be useful for debugging, or if end up finding a clean fix on the VFX graph side of things.

// from WorldManagerBhv.cs
        private void swapZones() {
            Zone rPrev = m_rCurrentZone;
            m_rCurrentZone = m_rAdjacentZone;
            m_rAdjacentZone = rPrev;

            if (PR_Defs.bZONE_REPOSITIONING_ON) {
                Vector3 vOffset = -m_rCurrentZone.getOffset();
                m_rAdjacentZone.adjustOffset(vOffset);
                m_rCurrentZone.adjustOffset(vOffset);
            }

            m_iZoneLayerSwitch01 = 1 - getZoneLayerSwitch();
        }

Now what?

Well, I added back the possibility to damage and destroy enemies. I think it's reasonable to also reintroduce the player health management, and have the enemies able to attack and destroy the player.

Before thinking about adding real "attacks", let's do what I did in the PoC: the enemies damage the player when they're very close.

As first step, I'm going to implement a test behaviour to send, via inspector, the input events to the player state manager. I did it for the enemies, as shown at the end of week 21, and I think I'm going to do it for each element that can be tested individually.

I started by duplicating the `TestEnemyBhv` to a `TestPlayerBhv` and changing it accordingly, but that showed an obvious opportunity for refactoring so that the common part of any test class is factored out to an abstract base class (`ATestBhv`).

The refactoring was easy, but a small issue remains: the `RTLocation` type I defined (wrapping a zone identifier and a platform identifier) can't be automatically serialized by Unity, so this is what I get into the inspector (I marked the problems with a red X):

Tomorrow I'll try to fix it implementing a custom "property drawer" or changing the definition of `RTLocation` so that it can work with the default serialization.

2024-08-15 - Player refactoring

Well, it looks like removing the `readonly` attribute from the fields of `RTLocation` is enough to make the default Unity serialization work and have the field shown in the inspector.

I don't like it, but I can live with it, and it was quick.

I could probably keep the `readonly` and customize the visualization in the inspector so that it is actually shown as a non-editable text field, and doesn't take multiple lines, but I won't spend time on a debug feature that will only be used by me. At least, not today!

About the use of `RTLocation` parameter in the `teleport` call activated by the ProButton: I will just expose two integers and convert to my custom data type inside the button code:

// from TestPlayerBhv.cs
        [ProButton]
        void teleport(int iZoneId, int iPlatformId) {
            RTLocation dest = new RTLocation(
                new Id<RTZone>(iZoneId),
                new Id<RTPlatform>(iPlatformId)
            );
            getStateManager().teleport(dest);
        }

And here's the updated panel.

During weeks 3/4 of advanced prototyping, as I explained at the time, I just ported the minimal code needed to have the player movement working. Then, when adding the body tracked avatar, I did a little more work in that area.

Now, finally, it's time to properly refactor the player management so that it fits the current architecture.

The movement and teleportation should keep working in the same way, and I will add new stuff to handle health and mana levels.

This will naturally flow into the handling of the "high level" game loop, because something will need to happen when the health level reaches zero. A game over? A restart from the latest checkpoint? A mandatory gameplay "rewind", Braid style? I don't know yet.

Anyway, let's get started.

I implemented the handling of the "input signals" (`adjustHealth`, `adjustMana`, `teleport`) so that they correctly affect the player state, and checked (in editor) that they work properly, by attaching the `TestPlayerBhv` to the player and using the buttons.

Now, I should be ready to add to the enemies logic the part which actually causes the health adjustment on the player entity.

Tomorrow I'm off for a day out, but I plan to make up for it on Saturday.

2024-08-17 - Enemy attacks

I'm going to implement the basic "energy leeching" attack I had in the PoC.

This should mean, essentially:

• add the logic which lowers the player health when the enemy is close enough

• update the presentation of the enemies so that they play another sound and change color while they're dealing damage

Let's go!

At least for now, in the enemy lifecycle state machine (discussed in week 21), I decided to simply split the `core` state into:

• `looking_for_player`

• `damaging_player`

What logic did I implement?

In the `looking_for_player`, the enemy simply steers towards the player. When the distance is less than a certain threshold, the state machine switches to the `damaging_player`, which applies the damage and also keeps steering towards the player, but with different acceleration, so that it feels a bit "sticky" to the target.

On the presentation front, I just had to handle the new state, setting the VFX color to the player cyan (as if they're stealing the player health) and playing a different sound effect.

I also had to fix the calculation of the "enemy to player" vector, which was not properly implemented and broke after disabling the zone repositioning a couple of days ago.

That's basically it! Let' complete the week with a short video showing two enemy attacking and affecting the health bar on the left wrist (shown around second 10):

Next week I have to catch up with a few tasks, so I'm going to take a break from Particular Reality.

I should be back at work the following week, so stay tuned!

While I announced Particular Reality at the end of September 2023, I started prototyping it a bit earlier, on August the 5th. This means that this week marked the first birthday of the project!

I think it's the perfect moment to summarize what I've been doing, try to draw a balance of the work so far, and write a few words about how I plan to proceed.

How much time did I work on the project?

As shown by the yearly "heat-map" above, in the past year I managed to do:

• 16 full-immersion days of "Proof Of Concept prototyping"

  • roughly 10 hours/day

• 23 weeks of part-time "Advanced Prototyping"

  • roughly 20 hours/week (4 hours/day, 5 days/week)

That's strictly development time working on the game and writing the DevLog entries related to that work, so not considering auxiliary tasks like:

• working on the game website and social media accounts

• writing "extra" articles (like this one you're reading now)

• preparing material exclusively useful for social media

• dealing with hardware/software problems preventing me to work

The "extra" articles required a very variable amount of effort. Some were written in a couple of hours, while others required a lot of work. In some case, the write-up itself was pretty fast, but was about a very time intensive task.

All things considered, not too bad but not great either.

Unfortunately, when other work or family related priorities arise, Particular Reality gets paused.

It's the price to pay for low-stress creative freedom, as discussed in a previous article.

Did I fulfil my own expectations?

If you're a developer, you're probably familiar with the awesome Hofstadter's Law:

It always takes longer than you expect, even when you take into account Hofstadter's Law.

Of course, this also applied to the development of Particular Reality.

I was quite happy of the results I got in only 16 days of full immersion, the phase I labelled "Proof of Concept Prototyping". For the next phase, the "Advanced Prototyping", the idea was reaching roughly the same features of the PoC, but properly engineering them, so that they can be extended and be the backbone of the full game.

If at the end of the PoC developement you would have asked me an estimate for this "Advanced Prototyping" phase, I would have probably guessed "about 3 months".

Instead, 23 weeks of development later, I'm not done yet.

And honestly, it's fine. I won't rush it. Successful estimates in software development are mostly a fairy tale anyway.

What did I get done, and what not?

I did some significant work on the game world management, going from a static, hardcoded demo map to an extensible, dynamic level management built around the idea of interconnected "zones" which can be designed indipendently.

The portal management itself was significantly improved from the version of the PoC, and I introduced the "bridge portals", which allow going from one zone to another.

Then, I introduced a player avatar, using upper body tracking (in the PoC I used "default", basic hand tracking), and did a lot of work related to the capture and playback of body motion, that I'm going to use for displaying "tutorial" gestures and for enemy animations.

I started working on a modular, extensible architecture that should allow me to load, save, pause and rewind the game.

Recently, I finally put back some enemies, implementing a wave management system and some other details that will allow me to have different kinds of enemies with different characteristics.

What's still missing, then, which was available in the PoC?

• it was possible to to shoot fireballs

• there was player health/mana management, and related pickups

• it was briefly possible (before I scrapped the feature from the PoC, to keep the scope limited) to pick up and use a melee weapon

I will proceed to add back in these features in the upcoming development weeks, each time trying to implement them in an extensible, robust way. For example, I don't just want to shoot fireballs, I want a subsystem that can activate different gameplay actions depending on the player equipment and gestures. Fireballs will just be one of many usages of such subsystem.

Did anybody care about all this?

So far, the project has not received significant attention.

The DevLog has only a few subscribers, and social media following is scarce too.

Still, there's a small set of loyal readers which make me happy, and some friends who support me with likes or reposting them to their audience. That's super-appreciated.

I hoped in a little more recognition, but I don't care that much. I know very well that there's an abundance of good projects out there, often better presented, and that the attention span of people online (myself included) is very short these days.

The core hook of the game should be its locomotion system, and that's something that is difficult to show in a convincing way to a mainstream audience. A handful of VR gamers and developers interacted with me enthusiastically about the concept, and that was refreshing.

I know very well that games catch people's eyes with cool visuals, before everything else, and the Particular Reality prototype graphics is currently awful: a bunch of placeholder elements and test colors picked almost randomly, with a couple of particle VFXs thrown in, a feeble attempt to push the imagination towards the right direction.

But I don't want to focus on the visuals right now: I want to complete the core gameplay systems first. I'm trying to play the long game, and I'm accepting that most people won't care at all about the project at its current stage.

It's also about not picking too many battles at the same time. I'm working hard to keep the gameplay logic separated from the presentation, so that I will be able, when the time comes, to focus on the visuals without breaking the gameplay (too much).

That said, I plan to change the most awful elements (like the red/green bridge portal *covers*) soon, provided it's something that doesn't take too much time. When I get to have all the PoC features implemented in the current version of the game, I'll capture new videos for the website and try to push them a little on social media.

Long story short: I'm not satisfied, but I'm not disappointed either. I will be disappointed only if the game will fail to gain any attention even when it will feature "real" graphics and audio.

Which might very well happen, but hey, let's hope not.

Meanwhile, if you want to push the project towards some kind of "social media respectability", you can follow its account on your favourite platform (or, even better, on all of them!).

And of course, if you're interested in articles like this one, consider subscribing to the DevLog.

Thanks for your support!

Year One Articles

To complete this year-long recap, I'm going to provide an "index" of the DevLog material I published so far, separating the "extras".

If this is the first time you run into an article of the DevLog, well, there's quite a lot of stuff you might want to read!

Extras

• Introducing "Particular Reality"!

• September 2023 interlude - The Unity fiasco, Quest 3 and more

• Coding conventions and Unity best practices - With a fair bit on the worst practices too

• How to pick a game project - Or, at least, how I did it

• Deep dive into body tracking - Tales from the Meta SDK

• A new development PC - Hardware and software setup

• The road towards a better DevLog - Featuring: a static site generator written in Rust

Proof of Concept Prototyping

• day 01: Setup, hexagonal grid

• day 02 - Basic input, target platform selection]

• day 03 - Teleporting between platforms (portal logic)

• day 04 - Teleporting between platforms (portal visualization)

• day 05 - Combat system basics

• day 06 - Hand poses recognition

• day 07 - Improving portals

• day 08 - Enemy basics

• day 09 - Sound effects

• day 10 - Health and mana

• day 11 - Fireballs

• day 12 - Health and mana pickups

• day 13 - Additional sfx, gameplay improvements

• day 14 - VFX research and testing

• day 15 - Switching to URP, enemies VFX

• day 16 - More VFX, wrapping up

Advanced Prototyping

• Week 01 - Getting started with level management

• Week 02 - Replacing the hexagonal grid

• Week 03 - Handling multiple zones

• Week 04 - Moving between zones

• Week 05 - Zones, portals and camera layers

• Week 06 - The color of the sky, the size of the play area

• Week 07 - Menu and mixed reality limbo

• Week 08 - When in doubt, use a state machine

• Week 09 - Refactoring player and portals

• Week 10 - Pause, Save, Load

• Week 11 - Revisiting zones and bridge portals

• Week 12 - Bridge portals done right

• Week 13 - Improving game world management

• Week 14 - A particular player avatar

• Week 15 - Simulation state history

• Week 16 - Motion capture and playback

• Week 17 - Motion clips management

• Week 18 - Additional clip management and debugging

• Week 19 - Clip playback on skinned meshes

• Week 20 - Return of the enemies

• Week 21 - Enemies life cycle

• Week 22 - Interacting with enemies

• Week 23 - Clearing Enemy Waves

As it surfaced in the latest entry with the "delayed hit" bug, I have some problem in my state management code.

Today I want to revisit the current execution flow, because there are too many details I don't remember well enough to debug it.

For starters, I'm going to check the data structures involved in the state management, and how they're related. I'm going to write a few notes, both for you and for future me.

Warning: this is not a suggestion about how to organize things... I'm just analysing the current state of the project, so that I can improve upon it: there are definitely improvements to be done on many fronts (naming, performances etc)

Let's start with a diagram:

Each one of these classes uses generic parameters so that it can be instantiated with custom data and logic. I'm skipping them to keep things short and simple. I just left in the diagram `D`, which is the custom data, and `S`, which is an enumeration of the state identifiers.

Obviously, the diagram is incomplete (only some data members are shown, and no methods are included): it would take too much time to be exhaustive, and it would not help to focus on the core aspects.

An `FSM` class contains the definition of the logic to be executed for each state, and a label (mostly useful for debugging).

• A single instance of a class inheriting from `FSM` can be used, of course, by multiple state managers on multiple execution acting on different data items.

`FSMExecData` contain some data common to every FSM execution, and a custom data element that changes depending on the FSM definition.

`FSMExec` contains an `FSMExecData` and some additional data, including an input queue.

• Why the split between `FSMExec` and `FSMExecData`? It's a clear separation between transient/runtime data and data that can be saved and loaded.

A class inheriting from `AStateManager` wraps a definition (`FSM`) and an execution (`FSMExec`) which operates on some data bound to the FSM (`FSMExecData`).

Every `update()` of a `AStateManager` object runs its logic and calls the `update()` methods of its child state managers.

• I like to keep a precise, deterministic and stable execution ordering. Never been a fan of the Unity approach using the script execution order to define what gets simulated first.

My time is almost over for today, so I'm just going to do a bit of clean-up and take note of possible problems. I usually add some short comments in the code, like `// !`, to mark something that needs further inspection, and then go and revisit the marked zones (using "find in files" or simply the git diff highlighting the changed lines).

2024-07-30 - More fixes, more problems

After yesterday's inspection, I feel ok with trying some changes.

While doing some more testing, I noticed that even keeping the history window to two elements only, sometimes a collision did not cause a proper hit. There's no delay, in this case the hit doesn't show up at all: I get the audio/wave feedback, but the enemy doesn't bounce.

I changed the update logic in the `EnemyStateManager` so that the movement of the enemy gets adjusted in a single point, after every force that can alter it gets accumulated. I also changed the global FSM execution moving it from `Update` to `FixedUpdate`, which should make more sense. Using `Update` should be limited to the _presentation_ layer behaviours, and should do some extrapolation of the data coming from the state data. But that's something to revisit at a later stage of development.

After these changes, it looks like the hits always register properly.

Then, I went and tried to understand the "history window" problem.

I found out that the `GameStateManager` did contain a separated `SimStateManager`, each wrapping a `SimFSMExec` initialized on one `SimFSMExecData` of the state history array.

That doesn't feel very sensible: a single `SimStateManager` should suffice, and should be able to set the (single instanced) `SimFSMExec` execution it wraps to the desired `SimFSMExecData` in the history buffer.

I guess I didn't want to change the `FSMExec` API while I was doing the save/load (which is when I drafted this, if I remember correctly), and as it didn't allow to change the wrapped `FSMExecData`, I worked around the problem doing this silly thing.

Having multiple `FSMExec` instances meant having different input queues, which probably introduced the "delayed" input problem discussed last week.

I changed the `FSMExec` API and eliminated the "multiple `SimStateManager`" anomaly.

Unfortunately, the fix doesn't work well. Other issues with other elements of the game appear. There must be some subtle issue I'm overlooking.

I think it's time for a strategic decision: I will temporarily disable the history buffer, and only use a single simulation state. I was working on the enemies, I want to keep working on the enemies.

Taking a significant detour to fix something that will be needed by other features, not yet developed, doesn't feel right.

With a single simulation state, it looks like everything works properly. Actually, remember the annoying "one frame glitch" that showed up when stepping into a portal? It's gone, so it's probably related to that too (and/or to other execution order fixes I did today).

To celebrate the unexpected good news, a short capture using a Quest build (instead of Meta Link).

You can notice a couple of glitches: the avatar VFX briefly disappears, and at the end, when the zone changes, the enemies visualization is somewhat off. We'll get there.

But tomorrow, I will start popping enemies.

2024-07-31 - Popping enemies

My goal for today should be pretty simple: handling the destruction of enemies.

I already did the heavy lifting in week #21, preparing and testing the full enemy life cycle.

So, it should only be a matter of sending to the enemy FSM the "pop" signal when appropriate, and everything should work.

Usually, enemies have an "health" value. Hitting them decreases such value.

The current health value can change the enemy behaviour, depending on the context and the kind of game. When the health value reaches zero, the enemy gets eliminated.

This kind of approach introduces a problem, which is making players aware of the current enemy health level. I don't want visible health bars (one of my design principles is "fully diegetic"), so I will need to find something, on the visualization side, which gives at least a rough indication.

In the debug visualization, instead, I can definitely use a health bar similar to the one I'm using on the player wrist.

Anyway, let's start introducing the "damage" logic.

To apply a damage, one should calculate the impact strength.

The simplest approximation that comes to my mind is considering the magnitude of the relative velocity of the "hitting" hand and the enemy.

Something like this:

public void applyHit(Vector3 vHitVelocity) {
    applyForce(vHitVelocity);
    Vector3 vRelativeVelocity = vHitVelocity - m_vVelocity;
    float fImpactForce = vRelativeVelocity.magnitude;
    float fHealthAdjustment = fImpactForce * fHIT_DAMAGE_FACTOR;
    m_fHealth01 = Mathf.Max(0f, m_fHealth01 - fHealthAdjustment);
}

In the enemy FSM logic, I have a state transition, from `core` to `destroying`, when the health value reaches zero.

After a bit of tuning of the `fHIT_DAMAGE_FACTOR` value, it looks like it's working properly.

I hoped to at least do the health bar visualization too, but something came up - I'm done for today!

2024-08-01 - Enemy health bars

First day of August! The Particular Reality birthday (August 5th) is approaching...

Today I'm going to add a debug visualization for the enemies health, and use it to make sure that the player hits affect it in a sensible way.

I can reuse the `p_energy_bar` prefab I already implemented for the player health/mana I'm visualizing near the wrists.

I just need to make sure that they are only visualized when in debug mode, and set the displayed value to the health value read from the enemy instance data.

Additionally, I need to rotate the energy bar so that it faces the player.

It's easy enough, so I'm going to take the chance to show a code snippet:

// skipped the "using" directives

namespace BinaryCharm.ParticularReality.DebugManagement {
    
    [RequireComponent(typeof(IStateManagerProvider<EnemyStateManager>))]
    public class DebugEnemyBhv : MonoBehaviour {

        [SerializeField] private Transform m_rDebugVisTr;
        [SerializeField] private EnergyBarBhv m_rEnergyBar;

        private void Update() {
            bool bIsDebugging = DebugManagerBhv.i().isDebugging();
            
            if (bIsDebugging != m_rDebugVisTr.gameObject.activeSelf) {
                m_rDebugVisTr.gameObject.SetActive(bIsDebugging);
            }
            if (!bIsDebugging) return;

            IStateManagerProvider<EnemyStateManager>[] rSMPs =
                GetComponents<IStateManagerProvider<EnemyStateManager>>();
            EnemyStateManager rESM = 
                rSMPs[rSMPs.Length - 1].getStateManager();

            StateManagement.Enemy.Data rData = rESM.getFSMData();
            m_rEnergyBar.setLevel(rData.m_fHealth01);
            m_rEnergyBar.gameObject.setRendered(rData.m_fHealth01 > 0f);

            BodyInputManagerBhv rBI = BodyInputManagerBhv.i();
            Vector3 vEnemyToPlayerDir =
                (rBI.getBasePos() - transform.position.withY(0f)).normalized;
            m_rEnergyBar.transform.rotation =
                Quaternion.LookRotation(vEnemyToPlayerDir, Vector3.up);
        }
    }
}

The only thing worth discussing in this snippet is the part where I fetch the `EnemyStateManager`.

I'm getting all the components implementing the `IStateManagerProvider<T>` interface, and using the last of the returned array as data source.

Why? Well, such interface is implemented both by `EnemyBhv` and `EnemyTestBhv`. Every enemy has a `EnemyBhv` attached to it, but if I want a "test enemy" which I can control via inspector, as shown in week #21, I just need to also attach at runtime an `EnemyTestBhv`and it starts driving the enemy, using a local `EnemyStateManager` detached from the whole game simulation.

So, the debug visualization uses the data coming from `EnemyTestBhv` if present, and from `EnemyBhv` otherwise.

Why didn't I directly use `GetComponent<EnemyTestBhv>()` and do `GetComponent<EnemyBhv>()` if the first call gave `null` as result? Well, `EnemyTestBhv` is not currently in the gameplay logic package, but in the main project `Assets`, because it depends on `com.cyborgAssets.inspectorButtonPro` (which I don't want to put as dependency of the gameplay logic package). The `IStateManagerProvider` allows to have a loose coupling. In a future refactoring pass, I'm going to separate the debugging scripts into a separate package, and at that point I will have no problem adding the external dependency.

The final goal, in terms of project/packages organization, is having a setup which will allow me to easily strip unnecessary elements (testing/debugging) from a build to be released. Not that I'm sure any of this prototype version will remain in the version that is going to be released someday.

After a bit of fiddling with scale and positioning, here's the result. I think it works pretty well: if I push gently there's a small health falloff, while if I hit more violently or two-handedly the falloff is significant, or even able to destroy the enemy in a single hit.

2024-08-02 - Changing wave, pausing enemies

Now that I can destroy enemies, it's time to take care of the "wave changes": in week #20, we assigned to each zone definition a number of enemy wave definitions. In week #21, we designed the FSM that should take care of the waves logic, which could be summarized by "materialize the enemies of each wave as long as the previous wave is cleared". We also took care of instantiating and destroying the enemies together with the zone that hosts them.

It's time to activate the logic "incrementing" the current wave counter when the current wave state is `destroyed` and see if everything works as expected.

Here's an excerpt from `ZoneStateManager`:

var rExecData = m_rFsmExec.getExecData().getData();
int iNumWaves = rExecData.m_rEnemyWavesFSMEx.Length;
if (rExecData.m_iClearedEnemyWaves < iNumWaves) {
    EnemyWaveStateManager rCurrWaveStateManager =
        m_rEnemyWavesStateManagers[rExecData.m_iClearedEnemyWaves];
    EnemyWave.EState waveState = rCurrWaveStateManager.getState();
    if (waveState == EnemyWave.EState.hidden) {
        rCurrWaveStateManager.materializeEnemies();
    } else if (waveState == EnemyWave.EState.destroyed) {
        ++rExecData.m_iClearedEnemyWaves;
    }
}

Cool, it works!

I still have some time, but I don't want to start big tasks at the end of the week. Let's see the TODO list:

• go back to the VFX graph stuff and see how to control the VFX simulation time, then bind it to my gameplay system

So, I don't want to touch the global VFX deltaTime setting, because I want to be able to control the simulation speed of different VFX elements in a different way.

After checking the documentation, I started remembering that I had already faced this issue while working on the portal VFX. when implementing the pause feature for the first time.

Unfortunately, it looks like the VFX graph doesn't allow to manually call an "update" of the particles simulation passing a custom delta time. That's very annoying.

The only workaround that I can find some traces about is disabling the automatic particle update options and replicate the logic myself, passing the delta time as a `float` parameter.

Here’s the panel with the Particle Update Options:

That shouldn't be too hard, but still annoying and I particularly hate the idea of having to do it for each VFX. Maybe there's a way to make a function and reuse it? I need to dig deeper. And I hate using node based editors.

I'm almost out of time, so for this week I'll be happy enough to apply the same hack I did in the case of the portal VFX: calling pause on the particles simulation. Yep, at least, the `VisualEffect` API allows to do that.

To end the week, a short video capture where I show the "pausing" of the enemies and the clearing a wave. At the end of the video, the next wave appears.

If you look carefully, around 00:23, there's an ugly glitching of the enemies particles. I think that's also due to automatic update in the VFX graph simulation.

When I change zone, as discussed long time ago, I move things around so that the current zone is always at the world origin. But the automatic particle update calculates the velocity on the basis of the world position change of the enemy, not its "logic" velocity (which is not affected by the offset I'm applying to move the zone).

I could work around that too passing the velocity to the VFX myself. Or, as it usually happens, I could spawn the first wave only after the player enters the zone, and not when it gets instantiated (which is when the player is on the bridge portal platform that acts as an entrance to the zone). But I kind like that you can see what awaits you on the other side of the portal!

Deep down, I know very well that the moment of the enemies appearance should be a design decision not influenced by technical limitations, so I'll end up fixing the problem anyway, because I know it's there and it would bother me even if not visible to the player.

Prepare for something special next week!

Hello again! Yes, I'm alive, and Particular Reality too.

Unfortunately, I had to take a pretty long break from the project development: my latest DevLog entry stopped at May 31. Almost two months! I feel awful about it.

That said, it's the harsh reality of this kind of "passion project" that, when other priorities arise, it is necessary to put things on hold.

But now, finally, I'm back, and hopefully I can stay on track for a while.

Now, were was I?

Before my (unwanted and unexpected) break, I was working on the enemies management.

More specifically, I had took care of the enemies creation and destruction, pairing it to the "zones" creation and destruction. But there was still a lot to do.

Let's check the TODO list from last time:

• go back to the VFX graph stuff and see how to control the VFX simulation time, then bind it to my gameplay system

• handle basic interaction with the enemies, so that the player can destroy them and clear waves

I'm going to start from the second task.

In the PoC, the player could destroy enemies by punching them (so, with a direct collision) and indirectly, by shooting a fireball.

We're not really focusing on combat techniques, and I'm not even sure I want the fireballs in the final game, so I'm going to keep it simple for now, and only allow destruction via direct hit.

Still, as we're now dealing with the "layered" architecture, it's going to be a little more complicated that what I did at the beginning.

Let's go back and see what I can "recycle", and what must be done completely different.

I'm doing (on purpose) a pretty basic usage of Unity, but I definitely want to at least use the collision detection system. This means having `Collider` components on the game objects in the scene, together with scripts with collision handler callbacks (that the engine will call when a collision happens).

The idea is that in my case these collision callbacks will do very little: essentially, record the event so that the simulation update can read it and take it into account in the update logic.

How does this fit the current architecture?

The core "problem" is that the presentation layer, by definition, should only render the state of the game. But the collider component and collision handling script are on the enemy game object, in the Unity scene, and so it comes natural to think about those elements as part of the presentation.

Ignoring that feeling and thinking about their functionality (detecting events happening in the simulation, and altering the state consequently), I can consider them as a source of "internal input" data to push towards the state machines of the "logic elements" these "scene elements" are bound to.

Specifically, let's say I update the position of an "scene enemy" by setting it to the "enemy state" position (read from the `EnemyStateManager`). If the new position triggers a collision, I can generate (and store) a logic event that I will fetch and process during the calculation of the following game state (shown in the next frame).

Until now, I have only been dealing with this kind of data flow (coming from the scene) for the player input (fetching the body tracking data). Now, I'm going to try to do the same with the enemies.

How to do that in practice? For today, I'm going to just read a bit of code I don't remember anymore, and tomorrow I'm going to start implementing the changes.

2024-07-23 - Implementing the collision data flow

Ok, a quick overview of what I plan to do before starting to write some code.

As I anticipated, the core point is that I have the state management separated from the presentation.

Let's consider a single enemy:

• it has an `EnemyStateManager`, implementing its logic, which operates on an `Enemy.Data` element (the part that goes into savegames)

• it has a subtree of nodes in the Unity scene, with an `EnemyBhv` on the subtree root.

`EnemyBhv` fetches the FSM execution state, and updates the scene nodes so that they visualize correctly such state.

Now, we need to have some data flowing in the opposite direction. Why?

As discussed yesterday, the collision detection happens in the scene, with Unity calling some callbacks when `Collider` components are properly setup on the relevant nodes.

When a valid collision happens, I will store its data into an input signal that will be sent to the `EnemyStateManager`, so that it can properly update the enemy state according to the collision data.

Ok, it took me a while (this is what happens when you stop working on something for almost two months!), but I almost got things working.

I added an `hit()` method to the enemy, and also updated the testing script that shows up in the video at the end of Week #21.

Using the testing behaviour, I can "hit" the enemy pressing a button in the inspector, and see that the enemy velocity changes because of the applied force.

So why I wrote that I almost got it working?

Wearing the headset and trying to hit the enemy with a punch, like I did in the initial prototype, does not work..

This means I have a problem in the part which actually detects the collision in the scene.

The problem might be on the enemy side, but could also be related to the hands.

I remember using a sphere collider approximating each hand, for quick and easy collisions. But maybe something broke while I was working on the "particular" avatar?

It's late: we'll find out tomorrow.

2024-07-24 - Restoring hand colliders

This is a good moment to make a copy of the repository pointing it at the end of the proof of concept development: while working on the new architecture and features, I had to break a good portion of it. Adding stuff is important, but cleaning up old cruft on the way is also necessary, so I removed or disabled many things. That said, having a working copy of the PoC can be useful, to use it as reference/guide while replacing subsystems and components with the new, improved implementation.

The "PoC" project works properly (you never know until you try), and I went in and checked how I handled the enemies/hands interactions.

Of course, something must always come up to interfere with working peacefully: Air Link doesn't work. I guess that some update broke something at some point in the last months. I already noticed it while working on another project, but I ignored the problem hoping it would fix itself with another update. That hasn't happened yet, unfortunately.

Irritating, but I don't want to waste the day trying to debug the issue, so I'll keep using the cable for a while.

As I guessed yesterday, while working on the player avatar I had skipped the invisible "hitter" spheres attached to the hands. Restoring them, the collision gets registered and the enemies react to the collision.

But something is off in the way the enemies move: there's a delay, and they go in the wrong direction. I need to dig deeper.

I went back to the PoC project and started porting some of the logic I implemented there, mainly the snippets handling enemy movement and its adjustment after a collision.

During the process, as I hoped, I had an idea about the odd behaviour I noticed while testing the new collision handling: I did not transform the "hit" vector (coming from the collision detection in the Unity scene) from the "scene" world space to the "zone relative" space that I use in the state management layer.

Additionally, I noticed that I don't keep track of the player position in such space: until now, I only needed the platform and zone identifiers. But now I need it, so I can steer the enemies not only towards the "player platform" but towards the exact player body position. No doubt that it will soon be useful in other contexts, too.

Tomorrow I'll try to take care of both issues.

2024-07-25 - From world to zone space

Following the intuition I had yesterday, I started coding and added a few methods to handle the transformation of vectors and quaternions from world space to the Zone local space.

Then, I used these methods in the "right place", which is the point of the code where the collision data recorded by the behaviour (on the scene node) is fetched and used to send an hit input to the FSM execution. This way, the "hit" event processed by the state management layer contains properly transformed data.

I also used this approach to steer the enemies towards the player, as I did in the PoC.

This time, I added a minimum distance check, to simplify testing: the enemies start going towards the player only when it's close enough.

This way I won't find myself submerged by all the enemies populating the current zone.

Well, to my surprise, things started working right after the code started compiling. At least in terms of positions and directions. I'll show you what I mean...

While the direction the spheres move towards looks reasonable, there's an insane delay from the moment I hit the spheres to the instant they react to the collision.

A one frame delay was expected considering the data flow I implemented, but this is way worse.

Tomorrow I'll revisit my steps and hopefully fix the problem.

2024-07-26 - Debugging the interaction delay

I'm going to think about possible causes for the delay in the interaction I observed yesterday and try to fix it.

As you know if you've been reading the past entries, I'm "all in" on visual debugging.

So, for starters, I visualized colliders for hands and enemies when in debug mode.

Then, I added back in the "collision wave" feedback to show hits, and I also changed the enemy particles color to white when a collision is happening. I'm cheating a little, because I'm managing such waves directly in the visualization layer. This would not work in terms of frame-accurate load or rewind system, but it's good for debugging.

I also capped the enemy velocity to a low value, so hitting an enemy doesn't send it too far away, and it can come back to be punched again.

While doing these changes, I also found that I had left both the "old" and new movement calculation (before and after porting the PoC code). That definitely didn't help.

This is the result: as you can see, the collision detection works properly, but the delay is still there.

My enemy "testing" behaviour currently only takes "hit" signals through the inspector buttons, and that works instantly, there's no observable delay.

Let's also bind the "hit" signalling to the hands collision and see what happens.

Ha! Now we're talking!

Now I know the enemy state manager works properly too, because the testing behaviour calls into it too. In theory, the enemy state manager update logic should be called every `FixedUpdate` both in the testing code and in the enemies that get spawned in a `Zone` according to the level description. But something is obviously wrong, because the non-testing enemies react to hits with a massive delay.

Testing again, I feel like the delay is around 2 seconds, which makes no sense. Unless... oh no, I think I know what's going on.

Yep, found it.

I don't remember if I wrote about this, but currently my state data is organized hierarchically in three layers, like this:

`AppState` -> `GameState` -> `SimState[]`

Basically, `AppState` contains some transient data and one `GameState`, which is what gets saved/loaded. `SimState` (which contains the player/enemy/world state of a certain instant) is the data structure that gets visualized by the presentation layer, basically representing a frame.

The `GameState` contains a "window" of a certain number of `SimState` structures implemented through a ring buffer. I constantly write to this buffer, overwriting old entries so that I only keep track of the latest `iSIM_HISTORY_SIZE` states. Currently, I had `iSIM_HISTORY_SIZE` set to 120... do you see where I'm going?

The current logic, when going from frame `x` to frame `x+1`, copies the previous state and then executes the update. But it doesn't copy the queued input, which is kept separated... and not cleared. So, it then gets processed after a full iteration into the `SimState[]` array has happened, which is a little less than two seconds later.

It's the first time I notice this because, I guess, it's the first time I queue an input to a state machine to the next frame. Yikes!

To verify that the problem is that, I just had to momentarily set

public const int iSIM_HISTORY_SIZE = 2;

...and guess what, "it works".

As it always happens, it looks obvious now, and I feel very dumb for not having thought about it instantly. But then again, after two months far from the project, I didn't even remember about this simulation state "window".

It's too late for a proper fix, but I'm happy enough to know exactly where to start on Monday.

I'm going to

• fix this state machine execution anomaly, a critical point of the whole game architecture

• proceed with the enemies interaction, handling their destruction so that I can finally test the "waves" system

And let's not forget I have another task from the previous TODO list:

• go back to the VFX graph stuff and see how to control the VFX simulation time, then bind it to my gameplay system

This week I'm super-busy, but I don't want to stop working on the project. I'll try to put in at least a couple hours every day, and maybe I'll do some extras on Saturday to balance things out.

That said: time to build upon the groundwork laid last week and spawn some enemies!

As previously discussed, in the "state management" layer, at least for now, it is the zone state manager which handles the enemy waves state update. I established that an enemy can't cross a zone boundary, so it felt the natural place in the data hierarchy of the game.

Consequently, in terms of instantiation and destruction of the enemies prefabs, it makes sense to similarly handle it together with the instantiation and destruction of the zone itself.

Will it be flexible enough to cover all cases the game will need? Probably not, but it should be quick to do, and it fits the current architecture.

I temporarily changed the wave definition for the `corridorZone` from two purple sphere to one purple and one green: this way, I don't need to reach the `rightTriZone` to test the spawn of different kinds of enemies.

Then, I successfully activated the instantiation and destruction of enemy instances, together with the instantiation and destruction of the platforms: I can go around the game world and the purple/green prefabs properly enter and exit the scene hierarchy.

There's a "small" problem: the enemies get spawned at the zone origin. Why?

I quickly figured it out: I was querying for the platform position from the zone definition in the presentation layer, which - obviously - is not ready yet, as I'm building the wave of enemies that will be part of exactly that zone definition.

So, I was bit by a default value returned if the actual value is not available, with a nice `// TOFIX` comment on the side. It turns out the `// TOFIX` comments don't help in avoiding errors if you never go and actually fix the problems you know are there!

I guess I know where to start tomorrow...

2024-05-28 - Fixing the spawn positions

Ok then, let's try to solve properly the enemy spawn positioning issue from yesterday.

Until now, I didn't really need the platform positions in the state layer.

They don't move, so I just calculated the proper offset of a platform on the basis of its coordinates, and set that offset when instantiating its game object.

This is different, because enemies will need to move around.

I could use the spawn definition to calculate the enemy spawn position, like I do for the platforms, but that fix would stop being useful as soon as I want to move them, which is definitely going to happen very soon.

So, let's do the right thing and start handling the enemy positioning from the state layer.

I added a `PosRot` in the `Enemy.State` structure, to store the zone-relative position and rotation of the enemy entity.

Conveniently, in the scene, I'm parenting the enemy `GameObjects` to the "zone parent" node. This way, I can set local position and rotation on the basis of the `PosRot` in the state, and everything should work fine even when I move the zones around (read the entries for weeks between 4 and 7 to know more about that).

I calculated the initial pose so that I can have a correct startup state, and the rest followed naturally from there.

Here's a screenshot showing a wave of enemies spawned on the correct platforms, as specified in the JSON definition on the right:

2024-05-29 - Getting rid of `ZonePlatformSettings`

There's something I "remembered" about the project while working on yesterday's fix which is bothering me a little.

I left as parameters defined in the code the radius of an hexagonal platform and the distance between two platforms.

Potentially, this would allow me to change those values from zone to zone, but I'm not currently defining these values in the zone description files: I have specific values hardcoded in `PR_Defs.cs`.

But I still pass the values around, which is a bit annoying when they could just be global constants.

I feel it's time for a design decision. Do I really want to have zones where these parameters change?

Honestly, I think I don't! The single platform radius must fit into the play area, or the whole concept of the game locomotion breaks. I could make them smaller, but that would make movement more cumbersome: you need to be able to easily step into a portal to go to an adjacent platform.

So, definitely not the radius change. What about the distance between platforms? That might be less problematic, but I don't see it adding any gameplay value. It could maybe change the aesthetics of zones a bit, but it could also be a bit confusing, and it would limit the kind of visual effects I want to add.

All things considered, I think I can lock these two values and simplify the code.

It's fine trying to be generic while things shape up, but I think there's value in periodically revising these decisions and trying to keep things simple.

Done: I got rid of the `ZonePlatformsSettings` structure and all its usages, simplifying the code in many places.

I guess today was one of those days where you just remove unnecessary code, which is always good (well, at least when nothing breaks during the process).

2024-05-30 - Enemy and EnemyWave state machines

While we want to have different kind of enemies with different behaviours, there's a core life cycle which can be shared by all of them: they appear in the scene, execute their behaviour (which could include roaming around, attacking the player, defending a platform etc) and, ultimately, get destroyed and disappear.

It's time to start working on this, because to be able to test the "wave change", I need to be able to destroy enemies.

Let's start with an important detail: a `GameObject` for an enemy entity that gets instantiated when we enter a zone, as we did on Monday, can be "active but doing nothing", and so it must be able to represent a `hidden` state: we can't assume that, when we spawn it, it must appear in the scene. Consequently, we must explicitly define and handle a `hidden` state as parte of the enemy life cycle. I'm keeping instantiation and destruction of instances confined to few, specific moments. This will be valuable in the future.

Well, do you miss my state machine diagrams? It's time to prepare one...

• an enemy instantiated but not yet "alive" is going to be in the `hidden` state

• when its wave gets activated, the enemy will transition to the `materializing` state, which will involve an animation

• when the animation is completed, the enemy will enter its `core` state, which for now could be an idle loop. At some point, in this state we're going to execute a nested FSM taking care of all the cool stuff an enemy will be able to do when active

• when hit, an enemy will get destroyed, transitioning to the `destroying` state, involving another animation

• when the destruction animation is completed, the enemy will reach is final state, `destroyed`

In my first draft, I sent the enemy back to `hidden` after `destroying`, but then I decided that it's better to keep them separated. They might visually be the same (enemy not visible), but logically speaking they're different: one is ready to enter in action, the other has been destroyed.

Additionally, they might not be visually similar: we might decide to show some remains for a destroyed enemy, especially while developing, for debugging purposes.

What about the FSM for a wave?

The structure should be pretty similar, but the logic a bit different:

I avoided using the spawn terminology and used materialization to emphasize that we are not creating the objects at that point, but activating them: creation and destruction is handled separately as we already discussed.

To complete the day, I implemented the two state machines into `EnemyStateManager` and `EnemyWaveStateManager`.

A "blind", untested implementation following the diagrams: tomorrow, I'll try to see it in action.

2024-05-31 - Enemy presentation layer

We instantiate and destroy enemies game in the scene according to the active zones, and we have defined a basic FSM handling their state updates.

It's time to do another important step: implement the presentation layer of an enemy, which basically means "setting" its `GameObject` so that it correctly represents what is described by its state.

To get started, I went back to the basic enemy implementation I did for the Proof of Concept prototype.

I used two simple procedural animations for the spawn and destruction of the "purple spheres", consisting of a scale change driven by an easing function.

Then, I have the particles VFX, and four sound effects:

• `enemy_spawn` for the materialization

• `enemy_alive` with a looping sound for the `core` state

• `enemy_pop` for the destruction

• `enemy_leech` used when the enemy was damaging the player

We don't need `enemy_leech` yet, but I definitely must use the other sounds.

I adapted the two simple animations and the sound playback so that they are driven by an `EnemyStateManager`.

The system works like this

• a presentation behaviour (`EnemyBhv`) on the enemy prefab is "bound" to an `EnemyStateManager` by setting the data needed to fetch from the simulation state a specific `EnemyStateManager` (what zone? what wave? what enemy in the wave?)

• during gameplay, that binding data is used to fetch the right `EnemyStateManager` from the whole game simulation

• the `EnemyStateManager` data is read and used to "configure" the prefab with a `set(EnemyStateManager rESM)` method

I don't remember how much I went into the details, but it's exactly how I already implemented the portals and the bridge portals "covers".

I remember that testing them was a bit annoying, so this time I'm going to do a bit of extra work to make debugging and tuning easy.

Ok, I'm pretty happy about the result: I modified `EnemyBhv` a bit and did a `TestEnemyBhv` that can drive `EnemyBhv` with manual input via the inspector.

When I add a `TestEnemyBhv` to an enemy `GameObject`, it disables the `EnemyBhv`, meaning that it doesn't run its `Update()` method (which fetches the `EnemyStateManager` of the enemy and calls `set(EnemyStateManager rESM)` to configure the `GameObject`).

Instead `TestEnemyBhv` hijacks the processing and calls the `set` method with its own `EnemyStateManager`, operating on test data accessible through the inspector.

So, via inspector UI, I can

• set any init state and data (as if they're coming from a save game), and initialize the execution with those settings, pressing "Reinit"

• inspect the current state data, and see how it changes following the FSM execution

• use some buttons to send the events that the FSM can receive (in this case, `Materialize` and `Destroy`), and check that the processing works as expected

• stop the normal, time based update, and execute the FSM one timestep at a time, pressing the "Step" button, or directly manipulate the state data (in this case, the animation progress)

Let's see all this in action with a short video:

This should cover all my needs, and I might extend it to save and load state configurations to test more complex situations.

With a little more work, I should also be able to generalize most of the test behaviour so that it works with anything built using my FSM-based gameplay system, but I'll think about that another time.

Going back to the enemy presentation, this is a good start but it's not perfect: the VFX update doesn't use my simulation time, which sucks.

But I'm out of time, so, let's write a TODO list for next week, and call it a day.

• go back to the VFX graph stuff and see how to control the VFX simulation time, then bind it to my gameplay system

• handle basic interaction with the enemies, so that the player can destroy them and clear waves

As soon as these two points are done, I will be ready to port to the new implementation the enemy AI from the PoC (basic movement towards the player, and energy leeching attack).

Until next time!

After the three weeks break I devoted to the static site generator, I really need a minute of planning and design before jumping into the code.

Today I only have a couple hours, so I'm going to focus on that.

First, let's write an updated TODO list.

• main game loop

  • improve the pause/unpause visualization and interactions (now we access it with the "system" pinching gesture, or with the test "left hand thumbs down" gesture)

  • implement (more or less explicit) access to the save/load system in VR (now we just use keypresses for testing)

  • implement the time rewind mechanic, which is also related to the "embodiment" or the detachment of the avatar (like pause/unpause and) which also is part of the pause/unpause and save/load features

• better avatar management

  • visual distinction for embodied/disembodied states

  • interaction between avatar and portals (when you stick an arm into a portal, it must not disappear but be different from the part which is outside the portal)

• better body mocap

  • possibility to show a reference skeleton coming from some selected keyframes, useful to capture looping and "connected" clips

  • possibility to tune the capture frequency and to resave a clip lowering it

• performance

  • add a pooling system to recycle dynamic objects

Ok, these are bullet points extracted from previous TODOs and I have also written a few new items.

I roughly sorted the items by putting the most important things first.

I definitely want to reach the point where I have a robust, complete main loop.

But I was thinking that, to have something interesting to distinguish the different states (both in terms of save/load that of time rewind), it's time to put back some enemies in the game.

In the initial "16 day" PoC prototype, for the enemies, I went for the minimalistic approach which was appropriate for that stage.

Enemies were simple spheres, and their behaviour was simply "float towards the player and collide with them to drain some energy". They had

• a collider the player could hit (directly with the hands, or shooting fireballs)

• two simple procedural animations for spawning and destruction

• VFX and SFX to make them a little more interesting

Everything was pretty much hardcoded. Waves with increasing number of enemies were progressively spawned according to a simple rule.

Now, it's time to set-up things properly, in a more flexible way so that I can spawn and handle different kinds of enemies, with distinctive behaviours and characteristics.

Let's try and put down a few ideas: I might change things at a later stage of development, but I need to start somewhere.

I think there are two core approaches that could suit the game in interesting ways.

One, more focused on exploration, could involve going around the game world progressively unlocking new areas, maybe with some puzzles and power-ups. In that case, enemies should respawn when entering a zone you have previously cleared, or going around the world would get boring pretty fast. There could be some "special" boss enemies that don't come back once you get rid of them, of course. This would be in some sense a "metroidvania"-like approach. Each zone would define a number of specific enemies to place at specific points, maybe adding a bit of randomness.

The second approach would be somewhat simpler, more linear: I would define for each zone some waves of enemies, and when the players clear them, they can access the following zone, never going back.

Waves could be defined by hand for each zone (even in that case adding some procedural/random variation, maybe). An excellent example of this structure, with lots of twists that made it more interesting, is Hades.

I like both approaches, but I'm a bit scared by the design complexity of the first one, and I feel like the second can work better with a more casual audience too.

But can I cheat and implement an enemies management system which could potentially serve both, and postpone high level design decisions?

Let's say that a set of enemies spawned at a single time is called `EnemyWave`.

A manager for these waves should take care of the specified enemies entities, both in terms of spawning and destruction.

I could make it more "active" or "passive", depending on how I want to wire it to the rest of the gameplay code.

Let's say I want to keep it "passive", it could have this API:

• `void spawnEnemies(EnemyWave rEnemyWave)`

• `void clearEnemies()`

• `bool isSomeEnemyAlive()`

Could this simple set of functionalities allow me to implement all the scenarios I'm considering?

Let's try and imagine how.

• Metroidvania-like approach: a `Zone` defines a single `EnemyWave`, describing the enemies that respawn (via a call to `spawnEnemies(rZoneWave)`) every time the player enters the area. Clearing the wave is not mandatory to proceed to a nearby `Zone`, so the portals "locking" is not related to the fact that the wave has been cleared or not. When a player exits, any enemies still alive would be automatically cleared calling `clearEnemies()`.

• Hades-like approach: a `Zone` defines one or more `EnemyWave`. When the player enters the `Zone`, the portals lock, and a counter of cleared waves (let's say `iClearedZonesCnt`) is set to `0`. Then, while that counter is less than the number of waves defined for the zone, if there are no alive enemies (checked calling `isSomeEnemyAlive()`), a new `spawnEnemies(rZoneWaves[iClearedZonesCnt])` is called, and the counter gets incremented. When there are no enemies alive, and no more waves to spawn, the exit portals unlock

• additionally, a survival mode: there's a single `Zone`, with an endless stream of new waves, which get generated as needed (when the manager detects there are no alive enemies)

This all sounds pretty reasonable, so I think that tomorrow I'm going to start implementing all this. I still need to think about how to define an `EnemyWave`, but I can definitely start with a basic definition and then improve it later, without even changing the method signatures.

For starters, it could be a list of pairs, where each pair is made of an enemy type and the zone location where to spawn them.

What about the enemy types? As first step, I'm going to put back in the spheres of the PoC, maybe doing a couple of variants. Then, I'd like to define a more sophisticated enemy type that makes use of the mocap features I implemented recently. In terms of behaviour, I'm going to define a set of basic features that can be composed to implement different "AI" for the enemies.

I'm going to define a perception system, or fake it (what does an enemy know about the world?)

Then I'm going to have a path finding system, so that an enemy can calculate how to reach a specific platform. And different kind of enemies could move in different ways (floating, jumping to an adjacent platform, teleporting....). What about attacks? Some enemies could only damage the player with a direct collision, while others might have ranged attacks, shooting projectiles or energy waves. Another interesting point might be having some sort of "group" behaviour when multiple enemies coordinate together.

There's plenty of possibilities, and in the spirit of prototyping, I'm going to try and implement many options and test them. Then, I'll decide which one work well together and get rid of the others

Tomorrow I'll start drafting the code.

2024-05-21 - Enemy waves definition

Yesterday I wrote a lot, and didn't touch the code. Today I'm going to do quite the opposite, starting to implement what I described yesterday!

I'm going to start with the data definition of the enemy waves, taking care of their persistence too, so that I can load them similarly to how I load a zone description containing a bunch of platforms arranged in a specific way.

I decided to keep the enemy waves definitions separated from the zone definitions, at least for now.

I added a simple enumeration in the global definitions of the game, to have a first, basic, enemy type identifier:

// from PR_Defs.cs
    public enum EEnemyType
    {
        purpleSphere,
        greenSphere
    }

Then I defined an `EnemySpawnDesc`, with information about the type of enemy and their spawn coordinates.

// EnemySpawnDesc.cs
using BinaryCharm.ParticularReality.HexGrids;
using Newtonsoft.Json;

namespace BinaryCharm.ParticularReality.LevelManagement {
    public struct EnemySpawnDesc {

        [JsonProperty("eEnemyType")]
        public readonly EEnemyType m_eEnemyType;

        [JsonProperty("hexCoords")]
        public readonly Hex m_hexCoords;

        [JsonConstructor]
        public EnemySpawnDesc(
                EEnemyType eEnemyType,
                Hex hexCoords) {
            m_eEnemyType = eEnemyType;
            m_hexCoords = hexCoords;
        }

    }
}

An `EnemyWaveDesc` is a simple aggregate of `EnemySpawnDesc`. I also added a string id, which might be useful while debugging (to distinguish specific waves).

// EnemyWaveDesc.cs
using Newtonsoft.Json;

namespace BinaryCharm.ParticularReality.LevelManagement {
    public struct EnemyWaveDesc {
        [JsonProperty("sId")]
        public readonly string m_sId;

        [JsonProperty("rEnemySpawns")]
        public readonly EnemySpawnDesc[] m_rEnemySpawns;

        [JsonConstructor]
        public EnemyWaveDesc(
                string sId,
                EnemySpawnDesc[] rEnemySpawns) {
            m_sId = sId;
            m_rEnemySpawns = rEnemySpawns;
        }

    }
}

Finally, I prepared a quick generator method to automatically prepare waves of enemies that could be suitable for a given `Zone`. This will help me with testing, considering that I don't have a level editor yet.

I pass in a dictionary asking to randomly put in the zone a certain number of enemies of specific types, and the method, provided there's enough platforms available (I also excluded the "bridge" platforms), assigns a random platform to each desired enemy instance.

I used a kind of Fisher–Yates shuffle algorithm: I put the identifiers of the suitable platforms into an array, and pick random identifiers from such array, progressively excluding them from the selection to avoid repetitions, so that I can only have one enemy per platform. I really like this algorithm as it has linear complexity, doesn't use extra data structures, and at the same time it's pretty intuitive. Intuitive enough that I haven't checked an "official" implementation but just wrote the following code, and it might very well contain bugs - you've been warned.

public class RandomEnemyWaveGenerator : IEnemyWaveGenerator
{
    public EnemyWaveDesc getWave(
            string sId,
            ZoneDesc zoneDesc, 
            Dictionary<EEnemyType, int> rEnemyTypeToCountMap) {
        int iTotEnemies = 0;
        foreach(var kv in rEnemyTypeToCountMap) {
            iTotEnemies += kv.Value;
        }

        PlatformDesc[] rZonePlatforms = zoneDesc.getPlaformsDescs();
        List<int> rNonPortalPlatforms = new List<int>();
        for (int i = 0; i < rZonePlatforms.Length; ++i) {
            PlatformDesc platform = rZonePlatforms[i];
            if (!platform.isBridge()) {
                rNonPortalPlatforms.Add(i);
            }
        }

        if (rNonPortalPlatforms.Count < iTotEnemies) {
            throw new System.Exception("Too many enemies for this zone!");
        }
        EnemySpawnDesc[] rSelected = new EnemySpawnDesc[iTotEnemies];


        int[] rSuitableIndices = rNonPortalPlatforms.ToArray();
        int iUsedPlatforms = 0;
        foreach (var kv in rEnemyTypeToCountMap) {
            for (int i = 0; i < kv.Value; ++i) {
                int r = UnityEngine.Random.Range(
                    iUsedPlatforms, rSuitableIndices.Length
                );
                int iRandomIdx = rSuitableIndices[r];
                PlatformDesc randomUnusedPlatform = rZonePlatforms[iRandomIdx];
                rSelected[iUsedPlatforms] = new EnemySpawnDesc(
                    kv.Key,
                    randomUnusedPlatform.m_hexCoords
                );
                int tmp = rSuitableIndices[iUsedPlatforms];
                rSuitableIndices[iUsedPlatforms] = rSuitableIndices[r];
                rSuitableIndices[r] = tmp;
                ++iUsedPlatforms;
            }
        }
        return new EnemyWaveDesc(sId, rSelected);
    }
}

2024-05-22 - Assigning wave definitions to zones

I could spawn wave definitions on the fly, but I prefer to take another preliminary step.

I want to generate some wave definitions for some of the zones in my current "game world", save them once, and then just load them. Exactly like I did with the zone definitions.

Having the enemies spawn consistently on the same platforms will be helpful while testing, and it will be definitely needed at some point by the "real" game: the only difference is that the waves are going to be defined through a level editor.

I defined another simple data type, collecting the wave definitions for a specific zone:

using BinaryCharm.Interfaces;

using Newtonsoft.Json;

namespace BinaryCharm.ParticularReality.LevelManagement {
    public struct ZoneEnemyWavesDesc : IHasStringId {

        [JsonProperty("sId")]
        private readonly string m_sId;

        [JsonProperty("rWaves")]
        private readonly EnemyWaveDesc[] m_rWaves;

        [JsonConstructor]
        public ZoneEnemyWavesDesc(
                string sId,
                EnemyWaveDesc[] rWaves) {
            m_sId = sId;
            m_rWaves = rWaves;
        }

        public string getId() {
            return m_sId;
        }
        public EnemyWaveDesc[] getWaves() {
            return m_rWaves;
        }
    }
}

I added to `PersistenceUtils` some utility methods to automatically save/load the `ZoneEnemyWavesDesc` objects to a specific directory, using their id as filename. Very similar to what I already did for the zones.

Let's define a few increasingly difficult "fights" for three zones of our test world:

• `corridorZone`:

  • `corridorWave0`: 2 purple spheres

• `leftTriZone`:

  • `leftTriWave0`: 3 purple spheres

  • `leftTriWave1`: 6 purple spheres

• `rightTriZone`

  • `rightTriWave0`: 1 green sphere, 2 purple spheres

  • `rightTriWave1`: 3 green spheres, 2 purple spheres

  • `rightTriWave2`: 6 green spheres, 6 purple spheres

The code to generate the desired waves is a (verbose) transposition of these definitions.

// called on startup, after loading ZoneDescs
static void WriteTestWaves(Dictionary<string, ZoneDesc> rZoneDescs) {
    RandomEnemyWaveGenerator rRandomWaveGen =
        new RandomEnemyWaveGenerator();

    EnemyWaveDesc rCorridorWave0 = rRandomWaveGen.getWave(
        "corridorWave0",
        rZoneDescs["corridorZone"],
        new Dictionary<EEnemyType, int>() {
            { EEnemyType.purpleSphere, 2 }
        }
    );

    EnemyWaveDesc rLeftTriWave0 = rRandomWaveGen.getWave(
        "leftTriWave0",
        rZoneDescs["leftTriZone"],
        new Dictionary<EEnemyType, int>() {
            { EEnemyType.purpleSphere, 3 }
        }
    );
    EnemyWaveDesc rLeftTriWave1 = rRandomWaveGen.getWave(
        "leftTriWave1",
        rZoneDescs["leftTriZone"],
        new Dictionary<EEnemyType, int>() {
            { EEnemyType.purpleSphere, 6 }
        }
    );

    EnemyWaveDesc rRightTriWave0 = rRandomWaveGen.getWave(
        "rightTriWave0",
        rZoneDescs["rightTriZone"],
        new Dictionary<EEnemyType, int>() {
            { EEnemyType.greenSphere, 1 },
            { EEnemyType.purpleSphere, 2 }
        }
    );
    EnemyWaveDesc rRightTriWave1 = rRandomWaveGen.getWave(
        "rightTriWave1",
        rZoneDescs["rightTriZone"],
        new Dictionary<EEnemyType, int>() {
            { EEnemyType.greenSphere, 3 },
            { EEnemyType.purpleSphere, 6 }
        }
    );

    EnemyWaveDesc rRightTriWave2 = rRandomWaveGen.getWave(
        "rightTriWave0",
        rZoneDescs["rightTriZone"],
        new Dictionary<EEnemyType, int>() {
            { EEnemyType.greenSphere, 6 },
            { EEnemyType.purpleSphere, 6 }
        }
    );

    ZoneEnemyWavesDesc corridorWaves = new ZoneEnemyWavesDesc(
        "corridorZone", new EnemyWaveDesc[] {
            rCorridorWave0
        }
    );

    ZoneEnemyWavesDesc leftTreeWaves = new ZoneEnemyWavesDesc(
        "leftTriZone", new EnemyWaveDesc[] {
            rLeftTriWave0,
            rLeftTriWave1
        }
    );

    ZoneEnemyWavesDesc rightTreeWaves = new ZoneEnemyWavesDesc(
        "rightTriZone", new EnemyWaveDesc[] {
            rRightTriWave0,
            rRightTriWave1,
            rRightTriWave2,
        }
    );

    PersistenceUtils.storeEnemyWaveDesc(corridorWaves);
    PersistenceUtils.storeEnemyWaveDesc(leftTreeWaves);
    PersistenceUtils.storeEnemyWaveDesc(rightTreeWaves);
}

Running this code gets me three JSON files in such directory, containing data that looks reasonable at a quick glance.

Tomorrow, I'll load these definitions and proceed.

2024-05-23 - Loading waves and initializing their state

So, on startup, where I actually load the zone definitions, I will also load the waves definitions.

Yesterday, by giving them the same string identifier, I established an implicit mapping from the `ZoneEnemyWaveDesc` to the `ZoneDesc` they refer to .

This is very basic and inflexible, but still a tiny bit better that saving the `ZoneEnemyWaveDesc` directly into the `ZoneDesc`: I can easily try different `ZoneEnemyWaveDesc` instances for a single `Zone` without having to touch the zone description.

You might remember from past articles that I'm only loading the "world" data definitions at startup, using to build a better representation to use at runtime (switching from `string` based identifiers to `int` based identifiers in the process).

Now, I augmented the `RTZone` (`RT` for "runtime") data type by adding an array of `EnemyWaveDesc`, which will be empty if there's no wave defined for that zone.

// rZoneEnemyWavesDescs is a Dictionary<string, ZoneEnemyWavesDesc>
// containing all the loaded ZoneEnemyWavesDesc 

EnemyWaveDesc[] rEnemyWavesZone;
if (rZoneEnemyWavesDescs.TryGetValue(
        sZoneLabel, 
        out ZoneEnemyWavesDesc rZoneEnemyWavesDesc)) {
    rEnemyWavesZone = rZoneEnemyWavesDesc.getWaves();
} else {
    rEnemyWavesZone = new EnemyWaveDesc[0];
}

rZones[zId] = new RTZone(
    sZoneLabel, 
    zId, 
    zd.m_sSkyboxMatId, 
    rPlatforms,
    rEnemyWavesZone // <--- new field
);

Crude, but ok for now.

We are done with the "static data" layer, now we need to go up in the architecture we have defined in the past.

First, we need to build the state management for the enemies, and finally we will have to implement the presentation of the state, managing the scene game objects which visualize the state.

The state management involves the dynamic data about enemies: the information which goes into a save game (and in the state history to be used by the rewind system).

I followed the path I already traced when implementing the zones/platforms.

I defined in the namespace `BinaryCharm.ParticularReality.StateManagement`:

• `EnemyWave` and `EnemyWaveStateManager`

• `Enemy` and `EnemyStateManager`

This hierarchical organization is pretty flexible, because I can define some update logic for a whole wave, and for each enemy.

I plugged an array of `EnemyWaveStateManager` into `ZoneStateManager`, together with the `PlatformStateManager` instances.

It looks like the more reasonable place, assuming enemies can't change zone.

Notice that since the `ZoneStateManager` update is executed for all the active zones (so, current and adjacent, if any), when I'm done you should be able to see enemies through a portal to an adjacent zone, as one would expect. Unless I want to spawn them after you enter the new zone for gameplay reasons.

There's quite a while of boilerplate to write to have the execution logic work in the layered architecture I established. In fact, I'm wondering if I should give some thought to simplifying things a bit.

Anyway, I can't complete this step today - fingers crossed for tomorrow!

2024-05-24 - State update, prefabs setup

I need to complete the state management, and then handle the instantiation and destruction of enemy game objects in the scene.

I'm going to keep things simple and just define two states for an enemy:

• hidden: the instance is ready but not displayed

• idle: the instance is displayed, and stands still on its spawning platform

Actually, I'm going to define the same two states for an enemy wave too.

I added the enemy waves state and a counter of the cleared waves to the `WorldZone.State` structure, that until now only contained the state data for the platforms in that zone.

In the state update, if the counter is less than the number of waves for the zone, I know I have an active wave, and that I must update the simulation of its enemies.

When all the enemies of the wave will get into the "dead" state, I will consider the wave itself cleared and increment the counter, switching to the next wave (if any).

But I currently don't have a dead state nor a way to destroy enemies, so my first goal is just being able to spawn the first wave of enemies for a zone, if any, at their correct spawn position.

I should have completed a basic implementation of the state management, and it's finally time to move to the representation layer, and put something on the screen.

For starters, I need to fetch back the enemy prefab from the PoC prototype.

I'm going to strip from it the behaviours that had to do with its gameplay logic (movement, interaction with the player) and just keep the functionality which is strictly correlated to visualization.

This separation of concerns adds a lot of complexity to the architecture of the game, but it will hopefully prove worthwhile, allowing me to do all kinds of cool stuff in a robust way. It's the kind of thing that separates a throwaway prototype, or a jam game, from the real thing.

I copied the old prefab and cleaned it up. Then, I duplicated it and made an alternate version changing the color to green.

I then added references to these two prefabs to my stupid simple resource manager, so that I can easily instantiate an enemy prefab of the desired type (specified by the `EEnemyType` enumeration).

I hoped to actually complete the wave instantiation, but there's still a lot of work to do. I need to "connect" the presentation layer to the state management.

As soon as I have the enemies correctly spawning according to the wave definitions, I will then proceed to add back in the possibility to destroy them. Otherwise, I can't "clear" a wave and get the next one to spawn. That should basically be my TODO list for the next week of development.

I'm no Knuth by far, but I decided to spiritually follow his tracks and take some weeks off the project to make something about how I publish the DevLog, hopefully improving the experience for both me and the readers.

Frustrations about post editing/management

There are quite a few things that annoy me about the current editing process:

• I write in Markdown, so I need to manually convert some formatting when I prepare a post for publishing using the Substack in-browser editor

• I don't have the level of control I'd like when inserting images and videos

• I can't use directly embed mermaid diagrams, which I find very useful. I'm forced to convert them to images, which is suboptimal

• I can't keep the "final version" of a post, with its resources, into a repository

Frustrations about content visualization

I find the Substack visualization good for generic text content, but not great for technical posts containing code snippets:

• there's no syntax highlighting

• code blocks wrap, while it would be better to keep the indentation and let the user scroll

• the fanciful HTML formatting of emails sent by Substack makes the code blocks unreadable (even when the code is less than 78 character columns wide, something which was fine even in text only emails as specified by RFC2822 a couple of decades ago)

• there are annoying popups

Frustrations about the state of the Internet

I thought that using a platform like Substack would help me reach a wider audience, but it doesn't feel like it is helping. Actually, I sometimes feel like it is damaging me: I keep reading about social media platforms sabotaging each other lowering the reach of posts including links, or links to content hosted on specific platforms.

Additionally, some people are actively avoiding Substack as a matter of principle, after some controversial content was not banned from the platform.

Do I agree with that? Do I implicitly endorse Substack by publishing content there?

Why do I even need to spend time considering these things?

Isn't developing a videogame and documenting the process stressful enough?

I don't care at all about the competition between tech giants, nor I want to be dragged into debates on freedom of speech and censorship.

I stay humble: maybe I'll give an opinion on "tabs vs spaces", on a good day.

Frustrations about other static site generators

I used Hugo for the Binary Charm website, and it wasn't a pleasant experience.

I quickly checked Zola and I felt like I would have hated that too.

I'm not saying they are bad, but I felt like, in terms of design, I wanted something different.

It was about controlling the processing logic and being able to define higher level data types on the source side. Basically, I wanted to do more processing inside the generator itself, and less on the "web" side of things. Template systems like Jinja allow to put a bit of logic into them, but they are limited, the syntax is awful, and debugging is basically trial and error.

So, around February 2023, I started working on a static website generator written in Rust, with the long term goal to handle the generation of my personal website, featuring a portfolio section done the way I wanted. I worked on it on and off, prioritizing other things (like Particular Reality, of course!).

The first website I published that used the generator was actually the game homepage, when I announced the project last year. A very simple "monopage" use case.

After some other months, at the beginning of 2024, I finally managed to add a first version of the portfolio management features, and I reached my initial goal of generating a new version of of my personal website, organized the way I wanted.

The generator was still pretty simple, with a few hacks and hardcoded things here and there, but did its job and I was quite happy about the result.

And, of course, it was something that gave me the joy of working the way I wanted, which should never be underestimated, especially when one is working on something personal with no strict time or budgeting constraints (that often force sacrifices and trade-offs on the technical side of things).

Extending my generator to handle the DevLog

The new stage I reached with my generator, paired with the unsatisfaction I discussed in the first paragraphs, pushed me to do another round of development, this time aiming to handle the DevLog entries, so that I can self-host them and fix all the issues I have with the current state of things.

As an added bonus, I will be able to port to the new system also some articles I wrote a long time ago, finally killing the Wordpress instance I am still forced to keep around at the time of writing.

Yes, Wordpress. Guess what? A dozen years ago I started using Wordpress for my website because it looked like the "wise" decision, the "not reinvent the wheel" approach, but even that time, I quickly realized that I hated it and that I did not really need it.

In my generator, every item is defined by a YAML file and a "side folder" with the same name.

The YAML file contains a "front-matter" with metadata about the item, and an optional body (a Markdown text).

The "side folder", if needed, contains one or more files making up the item. For example, an image resource side folder will contain a single file, while a video resource side folder will contain both the video file and an image, to be used as "cover" of the video when it's not playing.

The ability to define in the generator itself high level structured data (like a portfolio entry) is what makes it so convenient to use for me. I can define custom processing for each kind of item: for example, an image resource will produce multiple output images, at different resolutions, to automatically have a responsive image in the output. For video content, I'm currently embedding the Youtube version but I could automatically transcode the source video to a more compact, web-friendly format, and embed that.

To handle "articles", I revised the handling of the core resource type of the generator, the "page" one, which actually produces a web page in the website and can embed other "media" resources.

I had to add custom processing so that the Markdown text of an article can

• call "generator side" logic, via shortcodes

• handle syntax highlighting for code blocks

• handle Mermaid diagrams definitions

This is a video showing the generation in action at the current stage of development:

What's missing?

I still have to work on the DevLog "index" containing summaries for all the articles, and highlighting the latest entry. I must also add tags, something which I already did for my portfolio entries.

Then, I need to add the generation of sitemap and RSS feed. Yes, the RSS feed: it's time for their resurrection.

In visual terms, I'm not crazy about the colors used by the syntax highlighting and the Mermaid diagrams. I would also like to add support for a dark theme to the website, and add a dark/light switcher.

It shouldn't be that much work, and the hard parts should be done, but I decided to stop for now, and go back to the game for some weeks. Then, I'll take another "break" and come back to work on the generator, completing these last things.

What then?

I might change my mind, but the idea is port the old articles, a couple per week, to the new format, until the self-hosted version catches up. I might keep using Substack to publish a short summary of the articles, linking to the full length version, so I can still benefit from the newsletter and social functionalities offered by the platform. Or I might stop using it completely. We'll see how things go!

Meanwhile, see you in a week for some game development - much better than this web stuff.

After last week failure in modifying "blindly" the character rig to playback saved animation clips, let's approach the problem differently, ready to accept that it might be time to dig into the SDK code and understand it deeply.

I quickly checked week #14 to see which sample was my starting point, and noticed that, in that occasion, I refused to let the Project Setup Tool fix the outstanding issue "Body Tracking Joint Set should be set to Full Body".

And in those same days, I was wondering why the generative legs in the samples didn't work.

Silly me.

I let the Project Setup Tool apply the fix, and guess what? The legs started working.

Not that I need them, but it's good to know that the sample is running properly. Almost properly: the cup material is broken.

The scene I selected as starting point was `MovementISDKIntegration`, because it has the interesting feature of altering the input data to have the character bones follow some rules and constraints. While capturing this short clip, for example, in reality I'm completely closing my right hand in a fist, but the character hand adapts to a pose suitable to wrap the cup.

Not sure I'm going to use this, but it's interesting.

Now, let's see what the documentation said about this sample scene and the character rig.

While checking the documentation, I went back to the Unity-Movement repository and noticed that, just a few days ago, a new major release of the package (v5.0.0) has been released (my project currently includes v4.2.1).

This is a bit annoying: I was ready to start analysing the package code, but it doesn't make sense to analyse an outdated version. On the other side, if I update the package, I will almost surely need to repeat the steps I did last week to "clone" the needed elements from the Samples.

And to make things worse, to even try, I must also update the other Meta SDK packages, because this release depends on v63, and I have v62 in the project.

Boring! But it must be done. I really don't feel like doing it today, so I'm calling it a day a bit early.

2024-04-23 - Yet another SDK update

Surprise, surprise, I was on `v62` and `v64` is already available. I totally overlooked v63.

So, I'm going to switch the core SDK packages from `v62` to `v64`, and then I'm going to upgrade `com.meta.movement` from `v4.2.1` to `v5.0.1` - which was released... today! Glad I didn't install `v5.0.0` yesterday.

Let's upgrade everything and see if the project explodes.

So, my scene is broken as expected, and I assume it's because of the prefabs I duplicated last week, which should be re-duplicated or updated from outside the Unity editor.

There's something else which bothers me, tough: opening the sample scene I'm using as reference, `MovementISDKIntegration`, it looks like they introduced a problem with the hands of the robot character, which don't perfectly match the tracking.

Or is it because of something I did? It shouldn't be, but to be sure... let's test in an empty project.

I created an empty projects, imported the same SDK packages I'm using in the Particular Reality project, and tested the same scene.

The problem is still there - on the upside, the cup has the correct material.

This is quite annoying, because one would like to at least start from something working well before doing changes.

I verified that the problem only affects the robot character. If I open the `MovementHighFidelity` scene, the hands are correct.

I rolled back the `com.movement.meta` package to `4.2.1`, but keeping the `v64` upgrade.

Everything works again, which is nice: the `v64` upgrade doesn't break anything.

It's worth doing a commit just with the `v64` update, and then think about what to do with `com.meta.movement`.

As a side note, when I need to try out different versions of packages, my favourite way of doing is editing the `package.json` file directly. It's much faster than using the package manager, and using version control on the file I can quickly discard changes to rollback packages. Highly recommended.

So, the fact that a major release change breaks thing is pretty normal (that's usually the important difference between a major version upgrade and a minor version).

What options do I have?

• A) upgrade to `v5.0.1`, update my scene by re-cloning the sample prefabs, and ignore the hand tracking imperfection on the robot character I'm using for my tests

• B) stay on `v4.2.1`, analyse that and keep developing my things as if the new version doesn't exist

I was seriously considering option B, but then I started comparing the character rigs of `v4.2.1` and of `v5.0.1`, opening them side by side in two projects (the Particular Reality project rolled back on `v4.2.1`, the empty project from an hour ago on `v5.0.1`).

The setup in the new version is significantly simpler: it looks like the folks at Meta did some serious clean-up. This is very valuable to my eyes, so I definitely prefer upgrading.

I can live with the imperfect hands matching for a while, and with some luck, they're going to publish an update which fixes the issue before I even get to the point where I feel like I need to fix it myself.

So, let's update "my" prefabs and see if I manage to have my scene working on `v5.0.1`.

To only update what's needed, I compared (using WinMerge) the source files I duplicated from the samples last week with the same files in the new version of the package.

I found out that basically the only two files I need to update are

• `ArmatureSkinningUpdateRetargetSkeletonProcessor.prefab`

• `ArmatureSkinningUpdateRetarget.prefab`

So I'm copying the new versions of these files overwriting their "counterparts"

• `PRAvatar_SkinningUpdateRetargetSkeletonProcessor.prefab`

• `PRAvatar_SkinningUpdateRetarget.prefab`

...and I'm fixing their references to other cloned elements, using the `guid` replacement technique I discussed last week.

Ok, everything works again, and I'm now running `v64` and `v5.0.1` in my project.

I also introduced the hands displacement problem inherited from the samples, but that was expected.

Maybe tomorrow I'll be able to make some actual progress.

2024-04-24 - More visual debugging

Last week testing, when I used the "runtime gizmos", showed that the bones of the skinned mesh weren't properly rotated.

I wrote:

Everything looks working fine with the "skeleton" visualization because, I'm assuming, it doesn't really use the bones rotations, but just the joints positions, drawing lines between them.

After thinking some more, I think that assumption was incorrect, because I remembered that I can show the joints axis by setting on `PlayerAvatarDebugGizmos` an option inherited by `SkeletonDebugGizmos`.

And they are fine:

Good.

Now, let's give another look at everything and see if I can figure out why the skinned mesh animation is so broken.

While re-examining the scene, I noticed that the skinned character fetches data from an `OVRBody`, while the clip recorder saves data fetched from a `Body`. I forgot about this difference in the data flow.

Such script was attached to a node named `OVRBody`, which made things more confusing. Have I ever discussed my awful memory?

Anyway, I might have found the root of the problem. Fingers crossed. Let's recap and think about the body tracking data flow in the scene.

I discussed `Body` and `OVRBody` in my "deep dive" article.

Let's take the data flow diagram at the end of that article:

So, using a flowchart-like notation I'm making up on the spot, this is the current state of things:

Now, when I implemented `UpperBodyCharacterDataProviderBhv` last week, while fiddling with the data, I did actually somewhat revert the processing done in `FromOVRBodyDataSource`, but maybe I did something wrong there?

Last week I wrote:

I might hack around this, saving the data from the skinned mesh animation after the processing, but I don't really like that idea.

but in fact I'm already saving data that is "more processed" than it should be.

I'm saving the `Body DATA` into the clip, and not the "lower level" `OVRBody DATA`.

Basically, I could simplify the flow and avoid the "inversion" of the processing done by `FromOVRBodyDataSource` (that I'm currently doing in `UpperBodyCharacterDataProviderBhv`).

This is the plan:

I modified the data flow (partially: I didn't complete the "skeleton" path, which now is basically mirrored because I'm skipping the `FromOVRBodyDataSource` processing).

But... the `SkinnedMesh` behaves exactly as it did before. I guess my "inversion" of the `FromOVRBodyDataSource` processing was correct, after all.

So where's the problem?

Time for more visually aided debugging.

I'm going to do a test visualization based on the "runtime gizmos" we already used, but that uses the interfaces exposed by `OVRBody`, exactly like the skinned mesh rendering should be doing.

Here's my new debugging script:

using System;
using System.Collections.Generic;

using UnityEngine;

using Oculus.Interaction.Body.Input;
using BinaryCharm.ADT;

namespace BinaryCharm.ParticularReality.DebugManagement
{
    public class DebugAnimPlayerBhv : MonoBehaviour {
        
        private OVRSkeleton.IOVRSkeletonDataProvider m_rSkeletonDataProvider;

        private Dictionary<BodyJointId, GameObject> m_rJointToGizmoMap =
            new Dictionary<BodyJointId, GameObject>();


        private void Start() {
            initRuntimeGizmos();
        }
        private void Update() {
            updateRuntimeGizmos();
        }

        private void initRuntimeGizmos() {
            Action<BodyJointId> addRuntimeGizmo = (BodyJointId id) => {
                m_rJointToGizmoMap.Add(
                    id,
                    DebugManagerBhv.i().InstantiateGizmo(
                        null, PosRot.DEFAULT, 0.05f, "RG_" + id.ToString()
                    )
                );
            };

            addRuntimeGizmo(BodyJointId.Body_Root);
            addRuntimeGizmo(BodyJointId.Body_Hips);
            addRuntimeGizmo(BodyJointId.Body_Head);
            addRuntimeGizmo(BodyJointId.Body_LeftHandPalm);
            addRuntimeGizmo(BodyJointId.Body_RightHandPalm);
            addRuntimeGizmo(BodyJointId.Body_LeftArmUpper);
            addRuntimeGizmo(BodyJointId.Body_RightArmUpper);

            m_rSkeletonDataProvider =
                GetComponent<OVRSkeleton.IOVRSkeletonDataProvider>();
        }
        private void updateRuntimeGizmos() {
            OVRSkeleton.SkeletonPoseData skeletonPoseData =
                m_rSkeletonDataProvider.GetSkeletonPoseData();
            if (!skeletonPoseData.IsDataValid) return;

            foreach (var kv in m_rJointToGizmoMap) {
                int iJointId = (int) kv.Key;

                Vector3 vPos = skeletonPoseData.BoneTranslations[iJointId]
                    .FromFlippedZVector3f();
                Quaternion qRot = skeletonPoseData.BoneRotations[iJointId]
                    .FromFlippedZQuatf();
                kv.Value.transform.SetPositionAndRotation(vPos, qRot);                
            }
        }
    }
}

It's pretty simple. I spawn in the scene a bunch of runtime gizmos thanks to an utility method in `DebugManagerBhv`, one for each "key joint" I want to see. Then, I update their transforms on the basis of the `SkeletonPoseData` fetched from a script implementing the `OVRSkeleton.IOVRSkeletonDataProvider` interface that is expected to be on the same node.

In terms of our data flow diagram, this new visualization is based on `OVRBody DATA` like the `SkinnedMesh` character, instead of using `Body DATA` like the `SkeletonDebugGizmos` visualization does.

I decided to only show a few "key" joints, but with bigger gizmos. Less clutter, and easier to focus on the critical joints (root, hips, shoulders, hands).

Let's see what comes up!

Well, this new debug visualization tells me two important things.

First, I can now be 100% sure that the joints data provided by `UpperBodyCharacterDataProviderBhv` is correct.

Applying `DebugAnimPlayerBhv` on both character rigs, the one using the live data from the headset and the one using data from the recorded clip, gives consistent results, properly positioning the runtime gizmos on both.

Notice the runtime gizmos on my hands, at the beginning of the video, and how the gizmos on the character on the right move consistently with the (mirrored) skeleton on the left.

Second, I can see the difference in the pose of a runtime gizmo and of the actual bone of the `SkinnedMeshRenderer`.

I compared, using the Scene View of the Unity editor, the hips bone pose (using the editor gizmo) and the pose as calculated by the clip player (shown using the "runtime gizmo"):

Look at the two gizmos: not only the orientations are completely different, but the `SkinnedMeshRenderer` bone doesn't rotate at all.

Actually, not a single bone rotates, coming back to my intuition of last week: I thought that the skeleton visualization worked because it only needed positions while the rotations where wrong.

Then, I realized that it was using rotations after all. But I wasn't sure the data was correct, because I had two data paths with different processing. So, I changed that and became sure I had the correct data in place, with this new debug visualization.

At this point, I'm pretty confident that the issue is in the SDK code that takes the joints data and actually adjusts the bones.

Hopefully tomorrow I'll finally find the root cause.

2024-04-25 - Finding and fixing the bug

After the analysis conducted yesterday, I surrender to the idea of looking at the SDK code of the scripts involved in the character animation.

On the main node, there are these two scripts:

• `RetargetingLayer`

• `SkeletonProcessAggregator`

My idea since the beginning has been that these script access the skeleton data by calling the `GetSkeletonPoseData()` of the `OVRSkeleton.IOVRSkeletonDataProvider` interface.

It's not obvious where this happens (inheritance, partial classes...), so I'm going to set a breakpoint in my implementation of such method, in `UpperBodyCharacterDataProviderBhv`, and step into the execution, hopefully understanding why the code sets the positions of the bones but not the rotations.

After stepping for a while without noticing anything that looked problematic, I saw something that triggered my internal alarms.

Let's see the `SkeletonPoseData` returned by the `GetSkeletonPoseData()`:

    public struct SkeletonPoseData
    {
        public OVRPlugin.Posef RootPose { get; set; }
        public float RootScale { get; set; }
        public OVRPlugin.Quatf[] BoneRotations { get; set; }
        public bool IsDataValid { get; set; }
        public bool IsDataHighConfidence { get; set; }
        public OVRPlugin.Vector3f[] BoneTranslations { get; set; }
        public int SkeletonChangedCount { get; set; }
    }

To avoid some useless recomputations, at some points, the SDK code checks the `SkeletonChangedCount` field, and only acts if it changed.

But I don't remember setting this value in my implementation...

Ooops.

This was the code at the end of my `GetSkeletonPoseData()` method:

            return new OVRSkeleton.SkeletonPoseData {
                IsDataValid = true,
                IsDataHighConfidence = true,
                RootPose = rootPose,
                RootScale = 1.0f,
                BoneRotations = m_rBoneRotations,
                BoneTranslations = m_rBoneTranslations
            };

Unfortunately, I had not set at all the `SkeletonChangedCount` value.

Let's fix it incrementing a counter at each call and returning such value into the `struct`.

            ++m_iChangedCount;
            return new OVRSkeleton.SkeletonPoseData {
                IsDataValid = true,
                IsDataHighConfidence = true,
                RootPose = rootPose,
                RootScale = 1.0f,
                BoneRotations = m_rBoneRotations,
                BoneTranslations = m_rBoneTranslations,
                SkeletonChangedCount = m_iChangedCount
            };

The horror for the stupidity of the error, but also, the joy in seeing everything finally working!

A constructor would have forced me to initialize all the needed fields, but in this case the language couldn't support me in "not forgetting" to initialize and use that field, and a small distraction ended up costing me quite a bit of time.

Well, it's late, but this needs a celebration video where I record a clip and use it to high five myself.

I posted the last seconds on Twitter etc, but only you, dear readers, get the full story and the full clip.

2024-04-26 - Clean-up Friday

The las couple days have been a bit "heavy" and the weekend is around the corner, so I'm going to take it lightly and don't start implementing new features.

I want to do a bit of cleanup, because my code of the last couple days was mostly for testing and finding out what was wrong, and I took some shortcuts on the way.

Then, I want to capture a clip I can use for a cool "screenshot Saturday" update tomorrow (the last clip of that kind that I published is from more than two months ago, yikes!).

I removed the clutter and committed my changes to the repository.

I went back to the data path using `Body` data and not `OVRBody`, as it was innocent.

I'm doing a little more processing this way, but it feels cleaner as I'm operating on higher level data not using the Oculus vector and quaternion types but the Unity ones.

I left the other data path commented out in the code, in case I need to go back to it.

Then, I did a few minor changes to be able to capture a nice clip.

Namely, I:

• disabled the "skeleton" visualization unless debug mode active

• added a gesture to show/hide the mocap UI

• made the "playback" character invisible if no clip is playing

• fixed a culling bug that made the body vfx disappear once in a while

Then, I pushed a build to the Quest and recorded a motion clip in which I played the part of an enemy attacking the player as soon as they get on the platform. Then, I enabled the Quest video recording and "acted" the other part of the interaction.

It's just a glimpse into a possible future, and not a real gameplay interaction, but it should be cool for a promotional clip. The whole "behind the scenes" truth is just for my elite audience of DevLog readers!

Another week is over. How's things? Not bad!

I finally have a decent set of features related to body tracking.

The process started with the introduction of the `SkinnedMesh` avatar for the player character in Week #14, and now I can use that kind of character to playback motion clips, too.

I can conveniently record and edit those motion clips directly from VR, which is great.

Even if I'm not 100% done, I don't see major roadblocks in proximity. What's missing? A couple of minor things I already mentioned in the past:

• showing some "keyframe" references to be able to easily capture clips that can flow into each other

• being able to tune the clip frequency so that I can use clips with different "quality" as needed

I decided to postpone the implementation of these features to the moment when I'm going to actually start capturing and using clips.

I'm going to capture clips to act as "visual tutorial", showing the player the gestures they have to perform, and I'm also going to use them to prototype the enemies animations (movement, attacks, defences).

So what am I going to do, if not this, next week?

That's going to be a surprise, and it might take a little longer than a week.

Two weeks, maybe three. See you then!

I'm finally back at work on the project, after a three weeks long break due to other work commitments and to the setup of a new PC.

It always take a little time to go back to a project after a break, but unfortunately I can't avoid some interruptions. Let's start by copy/pasting the TODO I left myself at the end of week 17:

• I still need to add the tuning of the capture frequency, which shouldn't take long if I use the same UI control I used for the speed.

• I want to be able to visualize a reference skeleton when recording clips, to help in capturing looping clips

• I need to be able to playback a clip on a skinned mesh and not only using the debug gizmos (which only work in editor!)

• after doing that, I need to test everything with an Android build running standalone on Quest (I've been using Airlink for now)

I'd say that the capture frequency setup is not a priority. I want to capture on device, so I need to have everything running there, including the reference skeleton to capture looping clips.

All this implies that it makes sense to start from the third item of the TODO list: I need to be able to playback clips using as target a suitable skinned mesh, and not only the editor-only skeleton visualization.

I had already cloned the repository of the project and installed Unity during my PC setup last week, but I only had tested that it was running in editor, with no headset connected. Now, instead, using Oculus Link, something is broken: I have a never seen before exception thrown from `XRPass.cs`, a file part of the `com.unity.render-pipelines.core` package.

The fun begins (not). Do I have a problem with the new PC setup, did I forget to commit something to the repository, or what? Let's dig deeper...

On my old PC, I was working on the project using Unity 2022.3.19f1.

While doing my new PC setup, in a moment of optimism, I installed the suggested release, which was just a few minor versions ahead: 2022.3.23f1.

It is reasonable to install the "old" version to see if the problem is related to the new release (or maybe to some of the packages I'm using that needs to be updated too).

And, surprise: since the day I installed Unity on the new PC, there's a new minor release available, the 2022.3.24f1. With some luck, the new minor release could fix the problem. Worth a shot...

I installed Unity 2022.3.24f1, but unfortunately the problem was still there. I also updated some packages from the package manager. Meanwhile, I googled the exception, and found a thread about the same problem I was having.

Well, at least is not just me, I guess. There's no fixes/workaround suggested, and the post is from today (!). I could try to find a workaround, but I prefer to let Unity employees do their job, and focus on my game. Time to surrender and install exactly the same version of Unity I was using on the old pc, which is probably what I should have done in first place. Silly optimism!

Guess what, with version 2022.3.19f1, the exception disappeared. Yay.

Still, the "particular" player avatar doesn't show up. This looks more like an error of mine.

I went and compared using WinMerge the two copies of the project, on old and new pc.

This is exactly the kind of thing that makes me keep the old pc (and data) around for now, until I'm sure everything is in order on the new system.

I did find the problem, a couple of modified files in the Samples of the `com.meta.movement` package which I left out of the repository (around 800 Mb!). My bad for modifying those files directly, instead of making copies of them (placed in my packages) and modify the copies.

After copying the modified samples, the project works well (and runs significantly faster than on the old PC, which will be great to improve my iteration time).

When I thought it was over, the Oculus App prompted me for an app and drivers update. And... I have the v64 firmware update available on the headset too, which I'm actually quite curious about, since I heard it improves the passthrough quality.

I'm almost too scared to try if everything keeps working after the updates... should I leave it for tomorrow?

No, I need to know.

Pheew, everything worked. Oculus app updated, firmware v64 running on the headset, and the Unity project looks fine. I lost an afternoon, but maybe tomorrow I will actually be able to do some progress.

2024-04-16 - Removing the sample assets

I have very little time today, but let's see if I manage to get at least one thing in.

There's quite a lot of SDK code related to the animation of a skinned through body tracking data.

I'm talking about code in the namespace `Oculus.Movement` (from the `com.meta.movement` package).

If needed, I will analyse it in depth, like I did for the `Oculus.Interaction.Body.Input` namespace (from the `com.meta.xr.sdk.interaction` package) in the `Deep dive into body tracking` special post.

But first, I'm going to try to cheat and use (almost) "blindly" the player character setup which I already have in the Unity scene as a starting point. I'm going to try to clone it and just change the data source, so that it uses data I pass in (from an animation clip) instead of the body tracking data coming from the headset.

Before cloning the character, I want to remove the dependency from the Samples assets I discussed yesterday, because otherwise I would make it worse: I would duplicate something depending on assets which shouldn't be in the project in the first place.

This is more time consuming that it should be, because some dependencies can be not obvious.

Something particular annoying are the prefab hierarchies, like this one:

So, in the scene I have `ArmatureSkinningUpdateRetargetSkeletonProcessor`, but if I duplicate it in my package and rename it to `PRAvatar_SkinningUpdateRetargetSkeletonProcessor`, that duplicate will still have as variant parent the `ArmatureSkinningUpdateRetarget` prefab.

And that makes sense. But if want to proceed with my cloning, and duplicate `ArmatureSkinningUpdateRetarget` to `PRAvatar_SkinningUpdateRetarget`, there's no way (AFAIK) in Unity to "reparent" `PRAvatar_SkinningUpdateRetargetSkeletonProcessor` to `PRAvatar_SkinningUpdateRetarget`.

This is a case where I resort to violence, which mean editing the YAML of the prefabs to do the reparenting manually. The process is not obvious, so I'm going to describe it.

When you duplicate an asset, the duplicate gets a new `guid`. These global identifiers are used by Unity to handle references between assets. Thanks to this approach, one can freely rename assets without breaking stuff.

So, to do the reparenting, we must change, in the child prefab, the references to the parent prefab `guid`, replacing them with references to the "cloned father-to-be". You can find the `guid` of an asset opening with a text editor its `.meta` file.

Let's see, for example, the contents of `ArmatureSkinningUpdateRetarget.prefab.meta`:

fileFormatVersion: 2
guid: eb6bc10f92728a546a3d8c144d0ef787
PrefabImporter:
  externalObjects: {}
  userData: 
  assetBundleName: 
  assetBundleVariant: 

We also need to take note of the "cloned father-to-be" prefab, checking `PRAvatar_SkinningUpdateRetarget.prefab.meta`.

Done that, we know that

• `ArmatureSkinningUpdateRetarget` has `guid` `eb6bc10f92728a546a3d8c144d0ef787`

• `PRAvatar_SkinningUpdateRetarget` has `guid` `521e1b23a8494cd438c7a90ea68f6d82`

At this point, we can open with a text editor the child prefab, `PRAvatar_SkinningUpdateRetargetSkeletonProcessor.prefab`, and do a "replace all" that changes every occurrence of `eb6bc10f92728a546a3d8c144d0ef787` to `521e1b23a8494cd438c7a90ea68f6d82`.

After doing that, when I go back to Unity, I can verify in the inspector that the re-parenting has happened, so now we have:

I repeated the process to reparent `PRAvatar_SkinningUpdateRetarget` so that it points to `PRAvatar_SkinningUpdateTPose` and not `ArmatureSkinningUpdateTPose`.

As soon as I had my "complete" replica of the prefab hierarchy, I cloned the character and duplicated other referenced assets, so that the references point to the copy and not to the version coming from the Samples.

I put in the scene two character hierarchies: the original, and the new copy.

At this point, I can enable/disable the two `Character` and `PR_Character` nodes and they behave the same.

After a bit of supplementary trial and error for assets that I overlooked, I got my body tracked avatar working properly, and with no dependency from the Samples assets, exactly as I wanted.

The ultimate test was deleting the Samples folder of the movement package and test that everything was still working.

Here's my version of the assets. I'm not sure of the role of everything, but at lest now I know exactly which elements are in play.

Tomorrow I should hopefully manage to have multiple instances of my character rig running, with two different data sources activating the mesh animation (live tracking data from the headset, and recorded tracking data from a motion clip).

2024-04-17 - Playback on SkinnedMeshRenderer

Before wiring the different input data, I have to do a couple of minor things to prepare the character rig to exist in multiple instances.

First, I need to refactor the hack that I did a few weeks ago to scale down the legs to `0.01f`: that is currently done by `PlayerAvatarBhv`, which also handles other things like the mana/health bars.

I refactored the legs scaling moving it to a new `UpperBodyCharacterBhv` script.

Every instance of the character rig will have this behaviour, independently from where the animation data comes from.

Now, I want to be able to easily differentiate between the player avatar and other instances, and the obvious thing that comes to my mind is changing color to the particles that make up the body.

I added to the VFX of the body a parameter `ParticlesColorGradient` to be able to set the color by script.

The VFX node takes in a gradient, but being able to change the start color is sufficient.

This is the color change method I put into `UpperBodyCharacterBhv`:

        public void setParticlesColor(Color c) {
            var rColorKeys = new GradientColorKey[] {
                new GradientColorKey(c, 0f),
                new GradientColorKey(Color.white, 1f),
            };
            var rAlphaKeys = new GradientAlphaKey[] {
                new GradientAlphaKey(1f, 0f),
                new GradientAlphaKey(1f, 1f)
            };
            Gradient rGradient = new Gradient();
            rGradient.SetKeys(rColorKeys, rAlphaKeys);

            m_rVFX.SetGradient("ParticlesColorGradient", rGradient);
        }

Finally, it's time to start feeding to a second instance of the character some animation clip data.

The `PRAvatar_SkinningUpdateRetargetSkeletonProcessor` prefab has a pretty complex setup, with a bunch of behaviours storing quite a lot of settings.

I don't see direct references to `OVRBody`, so hopefully the other behaviours only access the tracking data through the interfaces it implements.

So, what I'm going to do, hoping that the Meta folks did things in a sensible way, is substituting the `OVRBody` behaviour, that I know fetches the tracking data from the headset, with another behaviour (`UpperBodyCharacterDataProviderBhv`) implementing the same interfaces.

The screenshot shows what my modified setup looks like, after collapsing all the attached scripts. Notice that I have removed `OVRBody` and added `UpperBodyCharacterDataProviderBhv`, which has empty stubs for the methods implementing the interfaces

• `OVRSkeleton.IOVRSkeletonDataProvider`

• `OVRSkeletonRenderer.IOVRSkeletonRendererDataProvider`

I'm now going to implement these interfaces and then add to `UpperBodyClipPlayerBhv` a reference to this new behaviour, so that it can update its data, like it already does with the `PlayerAvatarDebugGizmos` scripts that shows the debug skeleton we've been using to visualize recorded clip until today.

I implemented the interfaces and wired up the nodes appropriately, and it was finally time for testing. What happened?

Everybody loves videos featuring these kind of bugs, so here you are - I have no shame.

It looks like I overlooked something - it's not that strange, as the data expected by the two visualization differs. I have a pretty good idea of where to look to fix it, but it's late - I'll get it working properly first thing tomorrow.

2024-04-18 - Playback debugging

As it often happens, the problem shown in yesterday's video clip is harder to fix than expected.

I did some progress, but I also found a serious issue shat is shown by in the following clips, where I have attached two "runtime gizmos" to the hips bone of both character rigs, the blueish one for the playing character and the pinkish one for the clip playback.

The runtime gizmo I mentioned is a debug prefab I did a long time ago and always have around: you can make yours too in 5 minutes with zero 3D modeling skills, putting together three cones and three cylinders, and applying on them three unlit red/green/blue materials.

As you can see, the hips bone of the clip playback doesn't turn, while the arrow sticking out from my belly reasonably rotates according to my body position.

Everything looks working fine with the "skeleton" visualization because, I'm assuming, it doesn't really use the bones rotations, but just the joints positions, drawing lines between them.

Trying to use the same information to animate a skinned mesh, instead, fails miserably.

Additionally, the hands don't move at all, so there must be some other issue too.

I might hack around this, saving the data from the skinned mesh animation after the processing, but I don't really like that idea. It looks like I will need to dig deeper.

2024-04-19 - On-device serialization troubles

I doubt I could solve the issues I found out about yesterday in one afternoon, so I decided to end the week doing some other stuff.

This way, I'm going to tick off something from my list, and will postpone the more challenging task to next week.

Now that I have something, even if not correct, that should show up when running on Quest and not only in the Unity editor, I can make sure that the saving and loading of clips work on the headset too.

That's not to take for granted, as I'm using custom paths and file system access on Android has its own rules.

I tried running a build, and indeed, even if it doesn't crash and it plays back the latest recorded clip (that gets automatically set as "current clip"), the UI with the list of saved clips doesn't update.

Something is off, let's see if I can figure it out.

I don't remember if I already discussed the current save/load code, which is super basic.

Anyway, let's see the relevant parts.

        private const string sANIMCLIPS_PATH = "_animclips/";
	        
        private static readonly JsonSerializerSettings JSON_SER_SETTINGS =  
            new JsonSerializerSettings {  
                TypeNameHandling = TypeNameHandling.Auto,  
            };
        
        public static void storeAnimClip<H, T>(
                string sClipId, 
                AnimClip<H, T> rClip
            )
                where H : ICopyFrom<H>
                where T : ICopyFrom<T>, IInterpolable<T> {
            string sFilePath = sANIMCLIPS_PATH + sClipId + ".ac.json";
            store(sFilePath, rClip);
        }

        public static void store<T>(string sFilePath, T rData) {
            if (Application.platform == RuntimePlatform.Android) {
                sFilePath = Application.persistentDataPath + "/" + sFilePath;
            }
            string sDirPath = new FileInfo(sFilePath).Directory.FullName;
            if (!Directory.Exists(sDirPath)) {
                Directory.CreateDirectory(sDirPath);
            }

            Debug.Log("store to: " + sFilePath);
            string sJsonText = serialize<T>(rData);
            File.WriteAllText(sFilePath, sJsonText);
        }

        private static string serialize<T>(T rData) {
            string sRet = JsonConvert.SerializeObject(
                rData,
                Formatting.Indented,
                JSON_SER_SETTINGS
            );
            return sRet;
        }

Good: while on PC I just save close to the project files, on Android I remembered that I must save into `Application.persistentDataPath`, and I'm also logging the file path in `store`.

I tried saving a clip and then checked the game data path on device, which is:

/sdcard/Android/data/com.BinaryCharm.ParticularRealityPrototypeURP/files/

To access the Quest filesystem I use SideQuest.

I'm not sure why the Meta Quest Developer Hub only lets me access some areas of the device filesystem (in its File Manager window) and if there's any fixes for that, but well, SideQuest works, I'm going to use that.

Anyway, in SideQuest the data path contains an `_animclips` subdirectory, which is good news, but it's empty. Let's dig deeper...

This is one of those cases where attaching the debugger to the application running natively on an attached Android device (in this case, the Quest headset) is invaluable.

It kills iteration time, but if something works on PC and not on device, one needs to see what's going on, and that's the best way.

When I step into `JsonConvert.SerializeObject`, called by my `serialize` method, the control flow jumps out at some point. I know what that means: an exception got thrown.

Unfortunately, in these cases when you are running on device, you don't get the exception conveniently shown in the Unity console, and you need to search into the device logs, conveniently accessible through Meta Quest Developer Hub.

Here's the problem:

JsonSerializationException: Self referencing loop detected for property 'normalized' with type 'UnityEngine.Vector3'. Path 'rFrames[0].rJointPoses[0].vPos.normalized.normalized'.

I remember running into this in the past, but not exactly when, nor what I did to fix it.

After a quick google search, I remembered: this is one of the issues in the serialization of Unity types when using `Newtonsoft.Json` library.

But I added to my project the additional package that fixes these problems since the day I added any kind of serialization, exactly because I had already ran into the issue in the past. And it's working on PC. So what's going on?

I'm not even upset: is one of those frequent remainders that "write once, deploy everywhere" is and has always been not more than a fairy tale: in practice, there's all kinds of subtle platform differences that make things go sideways.

Summarizing, the problem should be that, on Android, the JSON library is not using the custom converters provided by `Newtonsoft.Json-for-Unity.Converters`.

The documentation says:

This package automatically adds all its converters to the `JsonConvert.DefaultSettings` if that value has been left untouched.

How does that work? I glimpsed at the code in `UnityConverterInitializer.cs` and there's a part using reflection to find the suitable converters that makes my spider-sense tingle. It's happened to me in the past that code stripping has thrown away needed classes that weren't explicitly referenced. Maybe that's what's happening?

But I won't dig deeper if I don't need to, and the documentation also explains how to explicitly initialize the serialization settings providing additional converters, and that sounds worth a try. By referring to them, I might also avoid the classes to be stripped from the build.

             new JsonSerializerSettings {
                 Converters = new JsonConverter[] {
                     new StringEnumConverter(),
                     new VersionConverter(),
                     new Vector2Converter(),
                     new Vector3Converter(),
                     new Vector4Converter(),
                     new Vector2IntConverter(),
                     new Vector3IntConverter(),
                     new QuaternionConverter(),
                     new ColorConverter(),
                     new Color32Converter(),
                     new Matrix4x4Converter(),
                     new LayerMaskConverter(),
                     new RangeIntConverter(),
                     new Hash128Converter(),
                     new BoundsConverter(),
                     new BoundsIntConverter(),
                     new PlaneConverter(),
                     new RectConverter(),
                     new RectIntConverter(),
                     new RectOffsetConverter()
                 },
                 ContractResolver = new UnityTypeContractResolver(),
                 TypeNameHandling = TypeNameHandling.Auto,
             };

It's not pretty, because I need to enumerate the classes, but it's also simple and explicit - the kind of thing which is perfect to see if something works.

And guess what? It worked.

Here's the first clip of the clip management running on Quest and not on PC.

Good, another small step forward has been done.

You can see how saving and loading is super-slow, locking the game for seconds, but that's fine for a development-only file format. In the final build of the game there won't be any JSON handling: at some point I will switch to tightly packed binary files and asynchronous file system operations.