For rigged animation
I suggest importing an existing rigged model. It doesn't really matter what it is. The reason for this is it's an easy way to get the correct object hierarchy:
As usual, there's a game_matrix and a RootNode that you don't want to screw with. The standard animations have an additional node, local_srt, that is a child of RootNode and an eventual parent of every other node. This node isn't
required, or really special in any way, but it makes things a lot more convenient - for example, moving/rotating/scaling the entire mesh would be a pain without it.
(Technically I lied about it not being special: GoromorgShieldComponent uses "local_srt" as the parent node for its visual effect. But most of your animations aren't for objects that use that component.)
Feel free to import a standard animation for this model, too, if it helps you think.
The mesh works as normal, just remember that it needs the RigModifier; don't remove or apply it. When you add a new bone, you'll need to add the vertex group for it manually, but that's easy enough. You also need to remember to add the
node for it manually; just add another Empty object and name it the same as the bone. Similarly, if you delete a bone, go ahead and delete the corresponding vertex group and node.
The node hierarchy should match the bone hierarchy. In-game, the node will follow the bone - however, Blender won't show that; the Empty object for the node will remain stationary while you play the animation. In the standard assets, the origin of the node always matches the origin of the corresponding bone (including rotation and scale). I recommend following this convention, it makes things easier.
Once you have your bones set up the way you want, go into weight paint mode and paint your weights as normal. Note that Grimrock will provide strange results (different from the results in Blender) if your weights aren't normalized, so I strongly recommend clicking Normalize All before baking your animation(s).
The rest pose will be used ingame until an animation plays. If you have a playOnInit or a MonsterComponent, the player will never see your rest pose, so it's just for your own convenience. (When an animation finishes playing, Grimrock doesn't return the model to its rest pose; it just leaves it in whatever pose the final frame of animation gave it.) But if your model is something like the giant gate in the standard assets, for example, which doesn't play an animation until it opens, then your rest pose matters. Anyway, at this point you can export your model - just remember to check the "Skinning" box in the exporter so your rig and weights actually get exported. (The exporter also lets you choose between exporting in the rest pose and exporting in whatever the current pose is, but since it takes 0.5 seconds in Blender to apply the current pose as the rest pose, this is a triviality.)
If you want to use multiple meshes, that's possible too. Look at assets/models/env/forest_oak.model; it should be pretty self-explanatory.
For the actual animation part, you can animate your bones as normal - just remember to insert keyframes. Don't worry too much about inserting keyframes on unnecessary channels (e.g. the "Whole Character" option) because you can just clean those up in the dope sheet later. Once your keyframes are done, go into Pose Mode and use Pose -> Animation -> Bake Action. Leave frame step at 1, and don't try to use visual keying or clear constraints/parents.
You could export the animation at this point but if you are like me, you want to optimize a bit. You may recall that Grimrock 2's animation format lets channels (location, rotation, scale) for each bone specify a single constant value, instead of repeating the same value every frame like Grimrock 1 animations had to. This really helps with animation filesizes since most bones have constant scale and location, for example.
Unfortunately, if you naively bake the animation and immediately export it, the exporter won't take advantage of this: when you bake a channel with a constant value in Blender, it gets two keyframes in the baked animation, one at the start and one at the end. When the exporter sees this, it sees there's more than one keyframe, so it assumes it can't treat that channel as a constant value, so it repeats the same value every frame in the .animation file, meaning that you get bloated animation files. You can check the dope sheet to confirm this.
You could also use the dope sheet to fix it, by manually deleting the culprit keyframes at the end - but that's obnoxious. There's an easier way: when you bake the animation, bake with an end frame to 1 higher than you actually want. Then delete that final keyframe entirely (select it and Alt+I). Then export. You'll find that the exporter has now taken proper advantage of this feature.
The exporter uses the frame rate specified in the Render panel (the camera icon in Properties; the same place you set antialiasing, aspect ratio, resolution, frame range, etc.). Most Grimrock animations are stored at 30 FPS, but any frame rate is supported, so go wild with that if you want to.
Object animations
These are much simpler, to the point of being almost self-explanatory - just import one of the standard assets' secret buttons and the corresponding animation, and you'll see nearly everything you need to do. There's no complicated object hierarchy, just use the regular hierarchy you'd have for a non-rigged and non-animated model. Indeed, you can make an object animation for any existing .model file without changing the .model itself.
Everything I said about baking rigged animations applies to baking object animations, including the final keyframe problem. Some object animations are so simple that you might prefer to make them frame-by-frame (e.g. pressure plates and most secret buttons).