Unsolved creating children in generator

I'm trying to create a plugin which behaves similiar to the Character module. I'll break it down to this:

• The plugin should be an object (using ObjectData).
• The plugin has settings for the user to modify.
• Depending on the settings the object will generate different objects.
• One of the settings (let's call it Mode) should toggle whether the user can see the generated objects as children of the plugin object (to interact with them) in the object manager or not. However the objects will always be visible in the viewport.

So my idea was this:

• In GetVirtualObjects I generate the desired objects based on the object settings. All desired objects will always be returned by this method.
• To add the objects as children to the plugin object in the document I have to use the main thread. For this I call ExecuteOnMainThread from inside GetVirtualObjects.

As I gathered from this thread with that lovely fella I am indeed allowed to create object hierarchies inside GetVirtualObjects, but I must not insert these generated objects in the document from inside this method (since it's not running on the main thread). So my idea for the code is this:

BaseObject* MyPlugin::GetVirtualObjects(BaseObject* op, HierarchyHelp* hh)
{
    Bool dirty = op->CheckCache(hh) || op->IsDirty(DIRTYFLAGS::DATA);
    if (!dirty)
        return op->GetCache(hh);

    BaseObject* generatedObjects = generateObjects(op);

    // Use a BaseLink to pass the current object to the method on the main thread as suggested in https://plugincafe.maxon.net/topic/13068/calling-executeonmainthread-from-objectdata-getvirtualobjects
    BaseContainer* bc = op->GetDataInstance();    
    BaseLink* link = bc->GetBaseLink(ID_PRV_BASE_LINK);

    if (link == nullptr)
    {
        link = BaseLink::Alloc();
        link->SetLink(op);
        bc->SetParameter(ID_PRV_BASE_LINK, link);
    }

    BaseDocument* doc = op->GetDocument();

    maxon::ExecuteOnMainThread([this, doc, link, generatedObjects]()
        {
            addGeneratedObjectsToDocument(doc, link, generatedObjects);
        });

    return generatedObjects;
}

void MyPlugin::addGeneratedObjectsToDocument(BaseDocument* doc, BaseLink* link, BaseObject* generatedObjects)
{
    BaseObject* op = static_cast<BaseObject*>(link->GetLink(doc));

    clearChildren(op);

    switch(getMode(link))
    {
        case Mode::ShowObjects:
            generatedObjects->InsertUnderLast(op);
            break;
        default:
            // Do nothing.
            break;
    }
}

So my questions before committing to writing the rest of the code is this:

• Is this the right approach for my goal? Should I call ExecuteOnMainThread from within GetVirtualObjects or am I on the wrong track?
• Is this the intended way to develop a plugin like this? Generators in the traditional sence take an input (which usually is whatever the user places as children of the generator) but in this case (just like with the Character module) the generator children will solely be managed by the object itself.

The code does work, I'd just like to make sure that I'm not messing something up by passing references the wrong way or by abusing the pipeline or anything like that.

Thanks for all you do guys!

Hello @cjthetiger,

welcome to the forum and thank you for reaching out to us. That is some pretty advanced stuff you are doing there for your first project, there are however a few problems with it.

1. You return your BaseObject* generatedObjects both as the cache result of your MyPlugin::GetVirtualObjects and try to insert it into the document your plugin node is contained in. You cannot do that. A node can only appear once in only one scene graph (i.e., document).
2. You are already aware about the threading restrictions, and are using ExecuteOnMainThread to defer your scene modification to the main thread, that is great!

There are however a few problems with this:

Blocking Behavior of ExecuteOnMainThread

maxon::ExecuteOnMainThread is blocking in the way you are calling it. You would have to call it with WAITMODE::DONT_WAIT to make it non-blocking. This is a problem because in your version you effectively tie MyPlugin::GetVirtualObjects to the main thread, i.e., your plugin is not really executed in parallel anymore. When there are 1000 of you plugin objects in a scene, they defer their load one-by-one to the main thread. The problem is that when you fix what with WAITMODE::DONT_WAIT, you lose the ability to react to something going wrong with your MyPlugin::addGeneratedObjectsToDocument.

Pointer Safety

There is also the problem that simply deferring things to the main thread won't solve the core issue. The reason why this restriction has been put up, is that Cinema 4D has an event loop as any other app and heavily relies on pointers (at least in the classic API). So, you could think of it like this:

STAGE1_ESTABLISH_POINTERS (main thread)
STAGE2_COMPUTE (parallel)
STAGE3_EXECUTE (main thread + parallel)
> repeat

So, when you are in STAGE2 of the event loop, it does not really matter where you invalidate pointers, main thread or not, it is always bad. The documentation for this was written before we had a ExecuteOnMainThread, so that is why they took an explanatory shortcut there. It is not about main thread or not, it is just that when you are naturally on the main thread, you are usually on what I dubbed here STAGE_1.

Adding objects to a scene graph outside of STAGE1 usually does not lead to crashes but adding objects can also cause other objects to be reallocated, and therefore invalidate pointers. The more dangerous thing you are doing is clearChildren(op);, which very much looks like a delete operation. If you do this from your GVO and the next evaluated node relies on pointers to these objects you just deleted, Cinema 4D could crash.

Okay, how do I solve this then?

You cannot, at least in the form you want. You must understand that the Character Animation tools just as Mograph are often very non-standard and not everything they do can be emulated in the public API (or is a good idea to emulate in the first place).

We had a long discussion about this in our morning meeting with different stances, but in the end, at least I would say, GVO is never a safe place to modify the scene graph of the attached document. No matter what you do.

The sort of old school 'hack' was to use ::Message to defer such operations until the method is naturally being called on the main thread and then do whatever you want. Depending on the message stream, this might take some time to happen, or in extreme cases, is going to never happen (very extreme case, NodeData::Message usually goes off very often). But this is IMHO still a hack.

The cleanest solution would be to add buttons to your object so that the user can click show and hide to expose whatever you want to be shown there in the Object Manager. But this would only solve the start and end point, not objects being replaced. So, to really tie this down, you would have to put more abstractions on top of it. Like for example implementing some MyPluginCacheReprObject which then links to an MyObject instance and starts inter-object communication via NodeData::Message to, broadly speaking, 'see what is what'.

This then can get all very hairy very easily. I personally would not do that. Caches do what you want to be done, they represent local dynamically modifiable hierarchies, but they are not meant to be shown to the user or referenced in other parts of the scene. I could write here more, but this is all getting very speculative.

1. I would recommend not following up with this general design, as it can lead to a lot of work and bugs.
2. I would rather recommend rethinking what you want to do. I do not know what you want to do here, but solutions could be using DescriptionCustomGui to show parts of other node descriptions inside your node and/or to split your functionalities into multiple nodes and let them interact as for example Mograph does.
3. But this can all get complicated, I personally, would simply embrace the limitations of an ObjectData plugin - you are not intended to modify the actual scene and mean to stay within the boundaries of your virtual scene, i.e., cache.

Cheers,
Ferdinand

Hey @ferdinand,

thank you very much for this in-depth explanation. And thanks to everyone involved for discussing this topic, it's very much appreciated.

My Plugin Idea

I'm creating an "Advanced IK Spline" with additional handles.

Demo Video

Demo Scene hosted on my personal OneDrive
The scene might not fully represent the screenshots below. I prepared this scene as a sort of blueprint which I'm now trying to re-create with this plugin during which I decided to rename some stuff and change the icon.

The generator object itself has various settings to control the generation of all required objects, consisting of some hidden joints, controller nulls, IK splines and layers.

The generated objects look something like this:

Note: The naming and some portions of the hierarchy is outdated in this screenshot.

The key features are:

• When the user chooses Mode "Setup" none of these objects should be shown in the object manager but in the viewport. Changing the settings will delete the previously created hierarchy and generate a new one which will match the new settings.
• When the user chooses Mode "Bind" only some Joints should be shown in the object manager so the user can select them and bind and weight paint a mesh to them.
• When the user chooses Mode "Animate" only some controller nulls should be shown in the object manager so the user can select them and animate the joints which are bound to them via IK spline tags.

The whole point of this project is to not having to look at all the behind-the-scenes objects which the user should not have to worry about. The user should only have to see what is relevant (in the viewport as well as in the object manager).

I'd like to give this dude credits for coming up with this rig while I'm at it:
https://www.youtube.com/watch?v=I4JL23DvaN0

In response to your answer

I think I understand all the points you're making and I'm really thankful that you're on top of your API game to help me to this extend.

While I do trust myself to create the code for proper caching in a stable manner I value your opinion more to respect the limitations of ObjectData. So that's what I'll do for now.

The question for me then is: How do I proceed? I could let the plugin create all these objects as intended in GVO, then make the user use Current State to Object or Make Editable to get all the virtual objects (while hiding behind-the-scene objects in an invisible layer) but this is not very user-friendly. The functionality I'd like to implement is best achieved with an approach similiar to the Character module imho.

My best bet then is to simply hope that you can come up with a better approach after reading my plugin idea. Until then I guess I will follow the approach I just described even though this might come with different kinds of problems in terms of usability.

What I would've loved most would've been to extend the existing IK Spline tag so the user can configure a handle to be influenced by the spline so it can act like the smaller handles you see in the demo video above. Maybe that's an idea for you guys for a future C4D update, I don't think extending existing Nodes is what us normal folks are supposed to do.

I have to bundle this whole thing and make it reusable in an easy non-confusing manner, that's the most important thing for me.

Best regards and thanks again,
Daniel

// EDIT
Also sorry for carrying this thread off-topic with my whole "but what should I do then?"-shenanigans. If you'd like I can create a new thread addressing this new issue.

Hey @cjthetiger,

Thank you for your update. This makes things much clearer for me. I would simply flip what you are doing on the head.

• You: Generate output O in GVO calls and then try to update tangible the scene graph in addition to returning O as the cache.
• Flipped: Update the scene graph when the user interacts with a parameter of your plugin and then copy these results when after that inevitable GVO is being called.

Besides doing the inverse, this approach also gives up the direct linkage of both operations:

1. User changes parameter (on main thread).
2. NodeData::Message is being called with MSG_DESCRIPTION_POSTSETPARAMETER for a relevant parameter.
3. You generate your output and modify the scene.
4. Cinema 4D invokes the cache being rebuilt because the data container of the node changed.
5. Your ObjectData::GVO is being called.
6. You copy what you did in 3.

There also come problems with this, as you might want to react to the user messing with the thing in the scene to reflect that in your cache. But that would have also applied to your case. I still do not really understand why you want to insert your result twice (once in the tangible scene, and once in the cache of your generator).

Find a quick sketch as a Python Generator object below.

Cheers,
Ferdinand

The result:
The scene: generator_scene.c4d
The code:

"""Demonstrates a pattern to safely modify the scene graph from a generator object.
"""

import c4d
import typing

# Stuff added by Cinema 4D
doc: c4d.documents.BaseDocument
op: c4d.BaseObject
hh: typing.Optional["PyCapsule"]

# The ID for the "Count" user data parameter. Normally one does not have to be that precise with
# id definitions, but in this case we are looking for a DescID in a message stream and if we want
# to be abstract in our code, i.e., do messageId == idIamLookingFor, idIamLookingFor must be
# constructed precisely.
ID_VERTEX_COUNT: int = c4d.DescID(
    c4d.DescLevel(c4d.ID_USERDATA, c4d.DTYPE_SUBCONTAINER, 0),
    c4d.DescLevel(1, c4d.DTYPE_LONG, 0))

# The ID for the "Segment Length" user data parameter.
ID_SEGMENT_LENGTH: int = c4d.DescID(
    c4d.DescLevel(c4d.ID_USERDATA, c4d.DTYPE_SUBCONTAINER, 0),
    c4d.DescLevel(2, c4d.DTYPE_REAL, 0))

def main() -> c4d.BaseObject:
    """ObjectData::GVO - This is just how the Python Generator object does it.
    """
    # The whole returning thins twice thing still does not make too much sense to me, I guess it is
    # about a node being copied without its children? I am just doing the inverse here of what you
    # did: I copy from the scene into a cache. The copying part is important. The question is also
    # what you want to copy here, I am just copying everything including tags, etc.
    null: c4d.BaseObject = op.GetDown()
    copy: c4d.BaseObject = null.GetClone(c4d.COPYFLAGS_NONE) or c4d.BaseObject(c4d.Onull)
    return copy

def message(mid: int, data: typing.Any) -> bool:
    """NodeData::Message - This is just how the Python Generator object does it.
    """
    # We get a message that one of the two parameters of our object that require rebuilding the
    # descendants of #op has been invoked. #op is the Python Generator object.
    if (mid == c4d.MSG_DESCRIPTION_POSTSETPARAMETER and
        data["descid"] in (ID_SEGMENT_LENGTH, ID_VERTEX_COUNT)):
        Update(op)

def GetSphere(ml: c4d.Matrix, label: str) -> c4d.BaseObject:
    """Yields a sphere object with the given label and local matrix.
    """
    sphere: c4d.BaseObject = c4d.BaseObject(c4d.Osphere)
    if not isinstance(sphere, c4d.BaseObject):
        raise MemoryError(f"{sphere = }")

    sphere[c4d.PRIM_SPHERE_RAD] = 25.
    sphere.SetMl(ml)
    sphere.SetName(label)
    tag: c4d.BaseTag = sphere.MakeTag(c4d.Tphong)
    if not isinstance(tag, c4d.BaseTag):
        raise MemoryError(f"{tag = }")

    return sphere

def AdjustChildren(root: c4d.BaseObject, newCount: int, segmentLength: float) -> None:
    """Adjusts the number of children and their position controlled by the controller object.

    I did not flesh this one out in all detail and mostly focused on being conservative with adding
    and removing nodes instead of nuking the whole hierarchy.
    """
    # Get the children of the null object.
    children: list[c4d.BaseObject] = root.GetChildren()
    childCount: int = len(children)
    delta: int = abs(childCount - newCount)

    # Try to be conservative with adding removing objects, when we nuke the whole hierarchy on every
    # update, this will break all base-links to these objects in the scene. We could here also be
    # more tricky and copy the ID_MAXON_CREATOR markers, so that links could be re-established, but
    # I just went with being conservative with allocation and reallocation here.

    if childCount > newCount:
        for _ in range(delta):
            children[-1].Remove()
    elif newCount > childCount:
        newMl: c4d.Matrix = children[-1].GetMl() if childCount > 0 else c4d.Matrix()
        for i in range(delta):
            GetSphere(newMl, f"{childCount + i}").InsertUnderLast(root)

    children = root.GetChildren()
    # Should not happen :)
    if len(children) < 1:
        return

    # Here you would have to do some curve fitting or whatever you want to do with your spline, I
    # am just lazy and put things onto a line :D
    for i, node in enumerate(children):
        node.SetMl(c4d.utils.MatrixMove(c4d.Vector(i * segmentLength, 0 , 0)))

def Update(node: c4d.BaseObject) -> None:
    """Called when the nodes managed by your generator object which are part of the tangible scene
    graph (i.e., non-caches) must be updated.
    """
    # Should not happen.
    if not isinstance(node, c4d.BaseObject):
        raise TypeError(f"{node = }")

    # Is essential, although Cinema 4D itself will always send description messages in the "correct"
    # context, nothing prevents a rogue plugin from bombarding you with SetParameter or the Message
    # calls in a dangerous context.
    if not c4d.threading.GeIsMainThreadAndNoDrawThread():
        raise RuntimeError("Scene modifications not allowed at this point.")

    # Get the two user data parameter values.
    count: int = op.GetParameter(ID_VERTEX_COUNT, c4d.DESCFLAGS_GET_NONE)
    segmentLength: float = op.GetParameter(ID_SEGMENT_LENGTH, c4d.DESCFLAGS_GET_NONE)
    if None in (count, segmentLength):
        raise RuntimeError("Could not access parameters for count or length.")

    # Establish the null-object below the generator.
    root: typing.Optional[c4d.BaseObject] = op.GetDown()
    if root is None:
        root = c4d.BaseObject(c4d.Onull)
        if not isinstance(root, c4d.BaseObject):
            raise MemoryError(f"{root = }")
        root.InsertUnder(node)

    # And update the scene.
    AdjustChildren(root, count, segmentLength)
    c4d.EventAdd()

Good evening @ferdinand,

my intention was not to create objects, return them in GVO and create clones of those objects to insert into the document. My intention was to create objects in GVO and insert the exact same objects in the document. During my prototyping with Python this actually worked; manipulating the objects in the document also manipulated the virtual objects. Obviously this is not what I'm trying to do anymore. Feels like this is something a teacher would schold me for.

Right now I'm thinking: Why use virtual objects in the first place? The only benefit of virtual objects in terms of usability is that I can prevent the user from manipulating them directly. Why not instead just insert all the objects in the document using the approach you described while limiting access to the user by manipulating the layers they're in? This way I wouldn't need any virtual objects anymore. Unless you tell me otherwise this is what I'm going to try to do in the next couple of days.

Then again I'm super tired and had a long day at work... Don't judge me for stupid ideas please.

Thanks a lot for the scene and the Python sample!

Best regards,
Daniel

Hello @cjthetiger,

my intention was not to create objects, return them in GVO and create clones of those objects to insert into the document.

I understood that, I added the cloning part as I tried to communicate in my code. As lined out in the first posting, nodes should not appear twice in a document (or even be attached to multiple documents). You do not pay the price here imideatly, as the cache is not yet part of the real scene graph, i.e., myCacheRoot->InsertUnder/InsertUnderLast is never being called. But it is also a dangerous thing for other reasons IMHO I am not going to unfold here in all detail. The cor idea is that GeListNode is mono-hierachical, i.e., a node can only have one parent.

This way I wouldn't need any virtual objects anymore. Unless you tell me otherwise this is what I'm going to try to do in the next couple of days.

I think this would make your plugin safer. And regarding this and also your "I trust myself to produce safe code" statement - you are working here for the pesky 1% of edge cases. Most of the time what you did above there will work, as nothing else is relying on these things. It is just for the 1% of "Well, you are not supposed to do THAT"-cases.

One last thing, because I saw your usage of layers, I am not sure if you are aware, but there are NBIT flags which can be get/set via methods attached to GeListNode. One of them is OHIDE and it will hide a tag/object in the Object Manager (but not in the viewport). So, if you want the user to give the ability to hide the children, you will not need a layer, you can just set OHIDE on the root object below your plugin instance. There are also the BIT flags which can be get/set with methods attached to BaseList2D.

Cheers,
Ferdinand

Hello @ferdinand,

sorry for keeping quiet, I'm rather busy these days.

I am aware of the OHIDE flag. But what if the user converts the object using Current State to Object? I wouldn't want invisible objects to stay in the document, the user should be in charge of handling them all once he takes this step. Just like when converting a Character object. So either I try to hook into the CStO command and remove the OHIDE bit from all objects or I keep using layers. Is there a reason to choose one over the other?

Thanks again for all the help you and everyone involved provided!

Best regards,

Daniel