Load BVMD Animation

Learn how to load and play animations in BVMD format.

Creae MMD Runtime

MMD has its proprietary animation system, so we provides a runtime to reproduce it. We will create an MMD Runtime and make the camera and mesh controlled by the runtime.

src/sceneBuilder.ts

const mmdRuntime = new MmdRuntime(scene);
mmdRuntime.loggingEnabled = true;
mmdRuntime.register(scene);

mmdRuntime.playAnimation();

• mmdRuntime.register(scene) - Register the runtime to the scene update loop. This is required to runtime to work.
• mmdRuntime.loggingEnabled = true - Enable logging. You can see some useful information (e.g. Animation binding failed bones) in the console.
• mmdRuntime.playAnimation() - Start playing the animation.

tip

It's possible to play animations even if no asset is loaded. In this case, the assets that are loaded later are automatically synchronized.

Now let's add objects that will be controlled by runtime.

src/sceneBuilder.ts

mmdRuntime.setCamera(mmdCamera);
const mmdModel = mmdRuntime.createMmdModel(modelMesh);

• mmdRuntime.setCamera(camera) - Set the camera to be controlled by the runtime.
• mmdRuntime.createMmdModel(mmdMesh) - Create an MMD model from the mesh. MmdModel is a kind of controller that abstracts and controls Mesh from the perspective of MMD.

Load BVMD Animation

For load BVMD animation, we use the BvmdLoader.

src/sceneBuilder.ts

const bvmdLoader = new BvmdLoader(scene);
bvmdLoader.loggingEnabled = true;

const mmdAnimation = await bvmdLoader.loadAsync("motion_1", "res/pizzicato_drops_yyb_piano_miku.bvmd",
    (event) => updateLoadingText(2, `Loading motion... ${event.loaded}/${event.total} (${Math.floor(event.loaded * 100 / event.total)}%)`));

• bvmdLoader.loadAsync(name, fileOrUrl, onProgress) - Load BVMD file. name is the name of the animation. onProgress is a callback function that is called during loading.

For handle MmdAnimation we need to import animtion runtime side-effect.

src/sceneBuilder.ts

import "babylon-mmd/esm/Runtime/Animation/mmdRuntimeCameraAnimation";
import "babylon-mmd/esm/Runtime/Animation/mmdRuntimeModelAnimation";

Here's how to add and play animations.

src/sceneBuilder.ts

mmdCamera.addAnimation(mmdAnimation);
mmdCamera.setAnimation("motion_1");

mmdModel.addAnimation(mmdAnimation);
mmdModel.setAnimation("motion_1");

• Both MmdCamera and MmdModel are designed to store multiple animations. Therefore, you must set the animation to use after adding it.

Change Animation Center

The motion that we use tends to move backwards as we go towards the end of the music. So you have to move the model and camera forward to keep them in the plane.

src/sceneBuilder.ts

const mmdRoot = new TransformNode("mmdRoot", scene);
mmdRoot.position.z -= 50;

// ...

mmdCamera.parent = mmdRoot;

// ...

modelMesh.parent = mmdRoot;

Make Directional Light Follow The Model

Our shadow frustum is fitted to the model. So we need to move the directional light to follow the model. to see the shadow properly.

src/sceneBuilder.ts

const bodyBone = modelMesh.skeleton!.bones.find((bone) => bone.name === "センター");
const meshWorldMatrix = modelMesh.getWorldMatrix();
const boneWorldMatrix = new Matrix();
scene.onBeforeRenderObservable.add(() => {
    boneWorldMatrix.copyFrom(bodyBone!.getFinalMatrix()).multiplyToRef(meshWorldMatrix, boneWorldMatrix);
    boneWorldMatrix.getTranslationToRef(directionalLight.position);
    directionalLight.position.y -= 10;
});

• modelMesh.skeleton!.bones.find((bone) => bone.name === "センター") - Get the bone named "センター" from the skeleton of the model. This bone is the center of the model.
• modelMesh.getWorldMatrix() - Get the world matrix of the model. Access after the Physics ensures that the matrix is up to date.

danger

For skeleton, which is being manipulated by MmdRuntime, the only way to get the world position value is to use getWorldMatrix() because the Matrix update method is overridden by MmdRuntime. (e.g. bone.getAbsolutePosition() doesn't work properly)

Because MMD uses its own matrix update policy, this flaw is inevitable.

Full Code at this Point

Here's the code up to this point:

src/sceneBuilder.ts

import "babylon-mmd/esm/Loader/Optimized/bpmxLoader";
import "babylon-mmd/esm/Runtime/Animation/mmdRuntimeCameraAnimation";
import "babylon-mmd/esm/Runtime/Animation/mmdRuntimeModelAnimation";
import "@babylonjs/core/Lights/Shadows/shadowGeneratorSceneComponent";
import "@babylonjs/core/Loading/loadingScreen";

import type { Engine } from "@babylonjs/core/Engines/engine";
import { DirectionalLight } from "@babylonjs/core/Lights/directionalLight";
import { HemisphericLight } from "@babylonjs/core/Lights/hemisphericLight";
import { ShadowGenerator } from "@babylonjs/core/Lights/Shadows/shadowGenerator";
import { loadAssetContainerAsync, appendSceneAsync } from "@babylonjs/core/Loading/sceneLoader";
import { StandardMaterial } from "@babylonjs/core/Materials/standardMaterial";
import { Color3, Color4 } from "@babylonjs/core/Maths/math.color";
import { Matrix, Vector3 } from "@babylonjs/core/Maths/math.vector";
import { CreateGround } from "@babylonjs/core/Meshes/Builders/groundBuilder";
import { TransformNode } from "@babylonjs/core/Meshes/transformNode";
import { Scene } from "@babylonjs/core/scene";
import { MmdStandardMaterialBuilder } from "babylon-mmd/esm/Loader/mmdStandardMaterialBuilder";
import { BvmdLoader } from "babylon-mmd/esm/Loader/Optimized/bvmdLoader";
import { MmdCamera } from "babylon-mmd/esm/Runtime/mmdCamera";
import type { MmdMesh } from "babylon-mmd/esm/Runtime/mmdMesh";
import { MmdRuntime } from "babylon-mmd/esm/Runtime/mmdRuntime";

import type { ISceneBuilder } from "./baseRuntime";

export class SceneBuilder implements ISceneBuilder {
    public async build(_canvas: HTMLCanvasElement, engine: Engine): Promise<Scene> {
        const materialBuilder = new MmdStandardMaterialBuilder();
        materialBuilder.loadOutlineRenderingProperties = (): void => { /* do nothing */ };

        const scene = new Scene(engine);
        scene.ambientColor = new Color3(0.5, 0.5, 0.5);
        scene.clearColor = new Color4(0.95, 0.95, 0.95, 1.0);

        const mmdRoot = new TransformNode("mmdRoot", scene);
        mmdRoot.position.z -= 50;

        const mmdCamera = new MmdCamera("mmdCamera", new Vector3(0, 10, 0), scene);
        mmdCamera.maxZ = 5000;
        mmdCamera.parent = mmdRoot;

        const hemisphericLight = new HemisphericLight("hemisphericLight", new Vector3(0, 1, 0), scene);
        hemisphericLight.intensity = 0.3;
        hemisphericLight.specular.set(0, 0, 0);
        hemisphericLight.groundColor.set(1, 1, 1);

        const directionalLight = new DirectionalLight("directionalLight", new Vector3(0.5, -1, 1), scene);
        directionalLight.intensity = 0.7;
        directionalLight.autoCalcShadowZBounds = false;
        directionalLight.autoUpdateExtends = false;
        directionalLight.shadowMaxZ = 20;
        directionalLight.shadowMinZ = -20;
        directionalLight.orthoTop = 18;
        directionalLight.orthoBottom = -3;
        directionalLight.orthoLeft = -10;
        directionalLight.orthoRight = 10;
        directionalLight.shadowOrthoScale = 0;

        const shadowGenerator = new ShadowGenerator(1024, directionalLight, true);
        shadowGenerator.transparencyShadow = true;
        shadowGenerator.usePercentageCloserFiltering = true;
        shadowGenerator.forceBackFacesOnly = false;
        shadowGenerator.bias = 0.01;
        shadowGenerator.filteringQuality = ShadowGenerator.QUALITY_MEDIUM;
        shadowGenerator.frustumEdgeFalloff = 0.1;

        const ground = CreateGround("ground1", { width: 120, height: 120, subdivisions: 2, updatable: false }, scene);
        const groundMaterial = ground.material = new StandardMaterial("groundMaterial", scene);
        groundMaterial.diffuseColor = new Color3(1.02, 1.02, 1.02);
        ground.receiveShadows = true;

        // create mmd runtime
        const mmdRuntime = new MmdRuntime(scene);
        mmdRuntime.loggingEnabled = true;
        mmdRuntime.register(scene);

        mmdRuntime.playAnimation();

        engine.displayLoadingUI();

        let loadingTexts: string[] = [];
        const updateLoadingText = (updateIndex: number, text: string): void => {
            loadingTexts[updateIndex] = text;
            engine.loadingUIText = "<br/><br/><br/><br/>" + loadingTexts.join("<br/><br/>");
        };
        
        const bvmdLoader = new BvmdLoader(scene);
        bvmdLoader.loggingEnabled = true;

        const [modelMesh, , mmdAnimation] = await Promise.all([
            loadAssetContainerAsync(
                "res/YYB Piano dress Miku.bpmx",
                scene,
                {
                    onProgress: (event) => updateLoadingText(0, `Loading model... ${event.loaded}/${event.total} (${Math.floor(event.loaded * 100 / event.total)}%)`),
                    pluginOptions: {
                        mmdmodel: {
                            materialBuilder: materialBuilder,
                            boundingBoxMargin: 60,
                            loggingEnabled: true
                        }
                    }
                }
            ).then((result) => {
                result.addAllToScene();
                return result.meshes[0] as MmdMesh;
            }),
            appendSceneAsync(
                "res/ガラス片ドームB.bpmx",
                scene,
                {
                    onProgress: (event) => updateLoadingText(1, `Loading stage... ${event.loaded}/${event.total} (${Math.floor(event.loaded * 100 / event.total)}%)`),
                    pluginOptions: {
                        mmdmodel: {
                            materialBuilder: materialBuilder,
                            buildSkeleton: false,
                            buildMorph: false,
                            boundingBoxMargin: 0,
                            loggingEnabled: true
                        }
                    }
                }
            ),
            bvmdLoader.loadAsync("motion_1", "res/pizzicato_drops_yyb_piano_miku.bvmd",
                (event) => updateLoadingText(2, `Loading motion... ${event.loaded}/${event.total} (${Math.floor(event.loaded * 100 / event.total)}%)`))
        ]);

        scene.onAfterRenderObservable.addOnce(() => engine.hideLoadingUI());

        mmdRuntime.setCamera(mmdCamera);
        mmdCamera.addAnimation(mmdAnimation);
        mmdCamera.setAnimation("motion_1");

        modelMesh.parent = mmdRoot;
        for (const mesh of modelMesh.metadata.meshes) mesh.receiveShadows = true;
        shadowGenerator.addShadowCaster(modelMesh);

        const bodyBone = modelMesh.skeleton!.bones.find((bone) => bone.name === "センター");
        const meshWorldMatrix = modelMesh.getWorldMatrix();
        const boneWorldMatrix = new Matrix();
        scene.onBeforeRenderObservable.add(() => {
            boneWorldMatrix.copyFrom(bodyBone!.getFinalMatrix()).multiplyToRef(meshWorldMatrix, boneWorldMatrix);
            boneWorldMatrix.getTranslationToRef(directionalLight.position);
            directionalLight.position.y -= 10;
        });

        const mmdModel = mmdRuntime.createMmdModel(modelMesh);
        mmdModel.addAnimation(mmdAnimation);
        mmdModel.setAnimation("motion_1");

        return scene;
    }
}