Solved Calling ExecuteOnMainThread() from ObjectData::GetVirtualObjects()

Hey guys,

I'm using C++, Windows 10, R23 and I've got a question about threading.

Currently my ObjectData plugin wants to guarantee the execution of some code during it's GetVirtualObjects() implementation to be on the main thread. For that I noticed there's ExecuteOnMainThread()
So the problem seems to be that when we are not on the main thread and we reach the call to ExecuteOnMainThread(myLambda, TRUE) asking it to wait for the execution, a job is enqueued but it is never executed, and we just hang. If we are on the main thread to begin with there's no problem. If we call ExecuteOnMainThread asynchronously there are no problems as well.
So is this expected behavior for GetVirtualObjects() or perhaps it is a bug?

Regards,
Georgi.

Hi @mastergog, the main issue is GetVirtualObjects is called within a Thread during the SceneEvaluation.
The scene Evaluation is responsible to iterate over objects/tags on the main thread and wait for each thread to be finished per priorities (this way we can ensure that an object with lower priority is correctly built otherwise you could run into huge issues. For the same reason in order to not screw the current Scene Evaluation, no other code is allowed to run in the main thread. No one wants to have its object deleted from a lambda executed on the main thread while your GetVirtualObject is currently executed within a thread.
So the main thread is blocked until all Jobs are finished, but in your case, you are also waiting for some code to be executed on the main thread so you run into the issue.

So either pass false to the wait argument or change your design.
May I ask why do you want to execute on the main thread? It's most likely that a Message approach is doable to achieve the same things.

Please let me know,
Cheers,
Maxime.

Hey @m_adam . Ok thanks for the info.

I was using the message approach with SpecialEventAdd(), which seems to be pretty much the same as using ExecuteOnMainThread() with wait==false
The reason for the routine is to create and insert a material the document, since I can't do that in a threaded context. The material is needed for a VertexColorTag that is inserted in one of my cache's objects.
Actually i'm still unsure if this is the best way to go about it, but it is the only that's worked so far. And everything seems to work so far, like loading saved scenes, deformerms, cloners, copying, undo, etc.

Hey @m_adam when exactly is the lambda executed when ExecuteOnMainThread() with wait==false is used.
Would it be right after the main thread resumes? The question is because I want to set some UI stuff on my BaseObject* in GetVirtualObjects() but the concern is the copy captured in the lambda might be invalidated in some fashion (although I have not been able to force it until now)

Thanks for your attention.
-Georgi

Hi @mastergog there is nothing using SpecialEventAdd().

The usual way before (ExecuteOnMainThread) was available was from the thread function call SpecialEventAdd, this will push a message into the CoreMessage stack. Then at some point (this really depends on what was pushed previously on the stack), your message will be sent. Then if you want to communicate with your plugin the usual way was to send another Message to this plugin.
So this gives us something like:
ObjectData call ExecuteOnMainThread -> MessageData (or any CoreMessage Receiver) do it's magic according the data provided -> MessageData (or any CoreMessage Receiver) call objectData->Message() -> ObjectData react to this message to update the UI.

ExecuteOnMainThread is kind of similar but doesn't use the same stack as the message stack, but instead use the Main JobGroup, then the lambda you provide is enqueued into this Job group and executed when it's its turn (see Jobs Manual.

Regarding the lambda, being invalid it shouldn't happen if you pass a copy and not a reference of this lambda. You can also pass a maxon::Delegate for the function to be executed later, due to the Maxon reference system, this will ensure the Delegate will be destroyed only when no more references exists.

Hope this helps,
Cheers,
Maxime.

@m_adam said in Calling ExecuteOnMainThread() from ObjectData::GetVirtualObjects():

Regarding the lambda, being invalid it shouldn't happen if you pass a copy and not a reference of this lambda

Hey I meant the BaseObject* object that my lambda captures by value (which is the parameter in GetVirtualObjects()). Can that get invalidated for some reason, before the lambda is executed?

Ha yes, the best way would be to pass a BaseLink hold by the ObjectData, and in your GetVirtualObject always update this BaseLink. You are not forced to make this BaseLink public you can just return it via a message like MSG_RETRIEVEPRIVATEDATA.

Of course in your lambda, you still need to check that your ObjectData instance is still alive :)

@m_adam
Hmm, I think I got confused a bit. Here's what i'm currently doing:
In GetVirtualObjects() there's this lambda:

GetVirtualObjects(BaseObject* object) {
    auto myLambda = [object]() -> void {
	if (object != nullptr) {
		MyObjectDataPlugin::copySettingsToUI(object);
	}
    }
    ExecuteOnMainThread(myLambda,false);
};

MyObjectDataPlugin::copySettingsToUI(object) is a static which calls SetAtomParameter<> with object for the different settings.

So this is not safe?

Not sure I got the idea with the BaseLink and checking if the ObjectData plugin is sitll alive.

EDIT: Also with the Message Data routine, if I send a copy of the BaseObject* pointer, is there the same concern it might get invalidated before the message is captured in MyMessagePlugin::CoreMessage()

Yes, you are right, but this is the only way otherwise you will send to the implementation (the ObjectData is the implementation of each instance of your object. That means in your case you are going to copy the setting to each instance of this ObjectData (this may be what you want or may not).
So yes using a particular instance may be invalidated but you can just check for nullptr, if the object is deleted why do you need to update it?

Could you describe the exact workflow you are aiming for? Updating the UI is pretty vague.
If you can share a code sample it will be constructive because for the moment the only thing I'm sure about what want to do. Is the creation of a Material from an ObjectData::GetVirtualObject. Then I don't know what do you want to do with this material or what do you mean by copy the settings to the UI (which UI).

Cheers,
Maxime.

Hey @m_adam thanks for the attention.
When I say copy settings to UI, I mean the custom user interface I have implemented next to the "Basic" and "Coord" tabs in the Attribute Editor.
So one of the workflows is -> GetVirtualObjects() does some stuff and it can change those settings and they need to be updated. In the previous post there's the code snippet about it.
I'm just unsure this nullptr check in the lambda is going to help me if the the object is actually deleted. Even if the lambda captures the pointer by reference, nothing would've made it a nullptr, as it is a parameter by value in GetVirtualObjects()

The material workflow (insert into document and also free) is different but essentially has the same issue with making sure that pointers provided by C4D, which my lambdas capture, are not invalidated.

Hi,

The UI is a Description, to customize your description this needs to be done via GetDDescription.
By Customizing I mean, modifying which widget is displayed, adding or removing a widget and I'm not sure is what you are aiming to do.
If you just need to set a value of a given gadget in your UI then call a SetParameter on the object instance and this should work.

Since the generator, is owned by the document, you are right there is no way to guarantee that the BaseObject representing will still be alive.
What you can do is passing a BaseLink* linking to your own Generator that your Generator is owning.
This way you can set it to nullptr safely (the reference in your lambda will also be nullptr this way) and BaseLink offers a way to retrieve the original AtomObject. For more information see BaseLink Manuel.
Addtionationally I would recommend inserting this BaseLink also in the Container of your Object. This way the document will acknowledge about your BaseLink and be able to update it in case of a Copy of it.
Otherwise, you will need to also override ObjectData::CopyTo to properly update the BaseLink/Invalidating it until GetVirtualobject is called.

But to be honest for me this is still a bit unclear, and until you provide a real example so I can reproduce your issue it will be hard to help you as I will be forced to stay abstract in the way I describe possible solutions since there are too many assumptions I need to guess.

Cheers,
Maxime.

@m_adam said in Calling ExecuteOnMainThread() from ObjectData::GetVirtualObjects():

If you just need to set a value of a given gadget in your UI then call a SetParameter on the object instance and this should work.

That's all.

What you can do is passing a BaseLink* linking to your own Generator that your Generator is owning.
This way you can set it to nullptr safely (the reference in your lambda will also be nullptr this way) and BaseLink offers a way to retrieve the original AtomObject. For more information see BaseLink Manuel.

Hey i've been trying to understand this but the manual hasn't been speaking to me... What do you mean by passing a BaseLink* to my Generator ?

As for a real example what more do you need than this minimal snippet ? There is an integer setting "MY_OBJECT_INTEGER_PROPERTY" that needs to be set, so some number is generated in GetVirtualObjects() and it is set. Btw this hasn't crashed for me yet, even though it was tested with lots of scenarios.

MyObjectDataPlugin::GetVirtualObjects(BaseObject* object) {
    int toSet = GetRandomNumber();
    auto myLambda = [object]() -> void {
	if (object != nullptr) {
		setAtomParameter<maxon::Int32>(*object, MY_OBJECT_INTEGER_PROPERTY, toSet)
		iferr_cannot_fail("Setting should not fail");
	}
    }
    ExecuteOnMainThread(myLambda,false);
};

Did you mean to allocate a BaseLink* with BaseLink::Alloc() before the lambda or have the BaseLink in the Description of MyObjectData plugin? Sorry for being a bit slow...

Hey @m_adam this is what I've tried:

GetVirtualObjects(BaseObject* object...) {
.
.
.
BaseLink* link = BaseLink::Alloc();
if (link != nullptr) {
	link->SetLink(object);
	auto myLambda = [&link, document]() -> void {
		if(link != nullptr) {
			BaseObject* object = static_cast<BaseObject*>(link->GetLink(document));
			if(object != nullptr) {
				// do something with object
			}
		}
	}
	ExecuteOnMainThread(myLambda,false);
}
.
.
.
}

But this seems to break easily in the link->GetLink(document) call when I copy my object a few times. Maybe that is why you mentioned overriding the ObjectData::CopyTo earlier? Or maybe i'm using the BaseLink wrong?

Regards,
Georgi.

@mastergog said in Calling ExecuteOnMainThread() from ObjectData::GetVirtualObjects():

But this seems to break easily in the link->GetLink(document) call when I copy my object a few times. Maybe that is why you mentioned overriding the ObjectData::CopyTo earlier. Or maybe i'm using the BaseLink wrong?

@m_adam said in Calling ExecuteOnMainThread() from ObjectData::GetVirtualObjects():

Addtionationally I would recommend inserting this BaseLink also in the Container of your Object. This way the document will acknowledge about your BaseLink and be able to update it in case of a Copy of it.

So doing something like

GetVirtualObjects(BaseObject* object...) 
{
	// First check if the Generator BaseContainer have a baseLink set
	BaseContainer* bc = object->GetDataInstance();
	if (!bc)
		return nullptr;
		
	BaseLink* link = bc->GetBaseLink(PLUGIN_ID);
	if (link == nullptr && link->ForceGetLink() == nullptr)
	{
		link = BaseLink::Alloc();
		if (link == nullptr)
			return nullptr;

		link->SetLink(object);
		bc->SetLink(PLUGIN_ID, link); // I use PLUGIN_ID since you want to be sure no one else will overwrite this value.
	}

	if (link != nullptr) 
	{
		auto myLambda = [&link, document]() -> void  ......

Cheers,
Maxime.

Hey thanks @m_adam . This seems to doing it for this workflow.

bc->SetLink(PLUGIN_ID, link); // I use PLUGIN_ID since you want to be sure no one else will overwrite this value.

So what if I have another thing I want to link. (PLUGIN_ID + 1) seems to be doing the trick. Would that be fine ?

This other thing is a BaseMaterial* instead of a BaseObject*, like so:

material = BaseMaterial::Alloc(Mmaterial);
if(material==nullptr) return;
BaseContainer* bc = object->GetDataInstance();
if (bc != nullptr) {
	BaseLink* materialLink = bc->GetBaseLink(PLUGIN_ID + 1);
	if (materialLink == nullptr || materialLink->ForceGetLink() == nullptr){
		BaseLink* materialLink = BaseLink::Alloc();
			if (materialLink != nullptr) {
				materialLink->SetLink(material);
				bc->SetLink(PLUGIN_ID + 1, material);
				auto insertMaterialLambda = [materialLink, document]() -> void {
				     // GetLink() for the material but it is always nullptr
                                     BaseMaterial* material= static_cast<BaseMaterial*>(materialLink->GetLink(document));
                                 }
                                maxon::ExecuteOnMainThread(insertMaterialLambda, false);
                         }
            }
}

It is as if BaseMaterial* can't be set to the BaseLink. The material that I get from the link in the lambda is always nullptr.

So in the end for the UI workflow this seems to work but it skips some of the steps, like linking to the BaseContainer*, you proposed in the last post. Capturing the links by reference was also causing the issues with the GetLink() calls I had mentioned and if lots of copies were made we'd crash in the BaseLink::Free() call.

BaseLink *objectLink = BaseLink::Alloc();
if (objectLink != nullptr) {
	objectLink->SetLink(object);
	auto copySettingsLambda = [objectLink, document]() -> void {
		if (objectLink != nullptr) {
			BaseObject* myObject = static_cast<BaseObject*>(objectLink->GetLink(document));
			if (myObject != nullptr) {
				// SetAtomParamter<> calls with myObject
			}
			BaseLink* toDelete = objectLink;
			BaseLink::Free(toDelete);
		}
	};

	maxon::ExecuteOnMainThread(copySettingsLambda, false);
}

And for the Material workflow the lambda captures a different BaseObject* myObject, which has a link in its description to the Material I want to insert in the document. I wasn't able to just link the material in the BaseLink.

BaseLink *materialLink= BaseLink::Alloc();
if (materialLink!= nullptr) {
	materialLink->SetLink(myObject);
	auto insertMaterialLambda= [materialLink, document]() -> void {
		if (materialLink!= nullptr) {
			BaseObject* myObject = static_cast<BaseObject*>(materialLink->GetLink(document));
			if (myObject != nullptr) {
                                BaseMaterial* material= getAtomBaseLink<BaseMaterial>(*myObject, MATERIAL_ID, document).GetValue();
                                if(material != nullptr) {
                                     document->insertMaterial(material);
                                }
                                BaseLink* toDelete = material;
			        BaseLink::Free(toDelete);
			}
		}
	};

	maxon::ExecuteOnMainThread(copySettingsLambda, false);
}

Do you think this is safe now compared to the original capturing of the BaseObject* object ? Stressing it a lot and it doesn't seem to crash (although it also wasn't crashing with the original solution)

Many thanks for the attention,
Georgi.

Glad you found a way and yes I think it's safe now :)
I guess there is no more open pint? If so I let you define your topic as solved (see Forum Structure and Features - Ask as Question If you don't know how to do it).

Just in case a BaseContainer can also store a BaseContainer. so you could store your own BaseContainer at the ID_PLUGIN And in this BaseContainer at ID 0 stores, the obj counts, and then you can safely iterate over it.

Cheers,
Maxime.