using c4d.threading with (python) thinking particles

Hello! I am curious if there are any gotchas to be aware of with using c4d.threading with thinking particles in Python.

Specifically, if I have two particle groups (let's say groupA and groupB), will each thread performing calculations on groupA that depend on groupB and making changes to either of these groups be able to see the results recorded simultaneously as stored in the document as the threads run through their set of assigned particles to calculate?

Let me break it down even further.

Let's say I have 100k particles in groupA that I am going to break up into 10k chunks across 10 threads. At some point in the animation, each particle is instructed to find a match with a particle in groupB to act as a unique target (think of the particle in groupA acting as a homing missile trying to land on a particle in groupB). When the groupA particle gets matched with the groupB particle, a bool switch stored in a TP data channel (matched) is set on the groupB particle to remove it as a potential match for other groupA particles. This way, there are no groupA particle overlaps when all the particles have reached their targets.

So essentially what I'm asking is... is this difficult to implement with threading? As each 10k particle count thread calculates, will they be able to see matches made via accessing groupB TP channel data from other threads as things are calculating? Will the TP channel data touched mid-process by one thread be seen by another thread? If not, is there some way to achieve this goal?

Hello @jenandesign,

Thank you for reaching out to us. Before we go here any deeper, I will just quote myself from the threading module overview:

Please note that C4DThread, just as its native Python counterpart threading.Thread, is a mechanism to decouple the execution of code, but not to speed it up. Its intended use is to make computationally complex tasks non-blocking for the main thread of Cinema 4D and by that ensure the responsiveness of the user interface.

Parallelism in the sense of executing multiple parts of a larger task in parallel is currently not offered by the Python SDK. This also applies by extension to the multiprocessing module of Python which offers such functionality, but it is not supported by Cinema 4D. Trying to use C4DThread to speed up the parallel execution of such multiple parts task will not only yield no execution speed improvements but will most likely even be slower than a single threaded execution of the same task.

In addition to that, Thinking Particles are single threaded themselves. So even if you would implement parallelized particle generation and update routines for them in C++, the core update loop of TP would still be single threaded which would make the whole thing a bit pointless.

The standard particle system is multi-threaded, but there are at least no easy ways to spawn standard particles on your own (you can technically manipulate the raw particle data tag, I showed once here how to deal with that tag and its data). The intended way to modify (but not spawn) standard particles is ObjectData.ModifyParticles.

Cheers,
Ferdinand

Thank you so much for the clear response, @ferdinand !

Oof... I have long been avoiding houdini, but I keep running into limits the deeper I go with particles in c4d python. Random things won't work in the TP python API (like trying to get/set a bool TP data channel among others), and the TP system as a whole feels like it hasn't been touched in a decade or maybe longer.

Perhaps procedural particles (controlled with python/xpresso) in c4d are in need of a rebuff? I hope the dev team is talking about it or considers the importance, because at this point a move to houdini seems like the only viable option from a production standpoint as particle counts go up.

I know this forum is mainly for plugin developers, and I appreciate you helping out anyway with this non-plugin-related topic. Thank you again

Hey @jenandesign,

Houdini is hard to beat at the moment when it comes to particles and simulation in general. But you could also give X-Particles a spin which is both performant and powerful.

The simulation tools of Cinema 4D have been a focus of the last releases and will continue to be a focus in upcoming releases. A new particle system based on the simulation framework is an item on our bucket list, but it might take some time until we get there.

Cheers,
Ferdinand

@ferdinand said in using c4d.threading with (python) thinking particles:

...
A new particle system based on the simulation framework is an item on our bucket list, but it might take some time until we get there.
...

That makes me very happy to hear, thank you for that!

X-Particles is very powerful and I have used it on many projects with great success over the years, but it misses the mark on UX within c4d due to the nature of it being third party. Working directly within and making use of all core toolsets in an integrated fashion is preferable when it comes to such delicate/intricate procedural tasks as particle motion. I have tried on several occasions to use X-Particles python API, but it as well suffers from severe performance restriction and a deluge of execution order mysteries- which always leads me back to just doing things in thinking particles. Simple.

I am very happy to hear that c4d might someday have a new particle system, though. That is terribly exciting, because in the meantime I now must consider facing the houdini mountain yet again while I wait with hope :cold_sweat:

Thanks again for your response- you've definitely inspired hope in me to stick around. I really love C4D, and I think you are doing great work with all of the recent improvements!

Hello @ferdinand! I hope this message finds you and your team well.

Sorry for the somewhat off-topic post, but I feel as though I must check back in after the last 2 weeks I have had.

I dove right into Houdini particles for about 10 days and now have a loose understanding of the layout of the program- at least enough to use it for work. I continue to feel tugged to learn more, but just today I came across a person on youtube, Dominik Ruckli, who has been releasing some incredibly deep tutorials using scene nodes that make C4D look like Houdini- even building his own particle setup!

After getting a lay-of-the-land in Houdini and now seeing the high level tool building potential in scene nodes, I can't help but think that C4D is actually not that far off from being able to match Houdini in a lot of ways.

I hope that the new simulation based particle system you've mentioned is in-part integrated into scene nodes workflow, because the vast theory and knowledge of Houdini artists would be almost drag and drop back into C4D (as can be seen with Dominik's work on a few concepts/setups). So much potential.

The future of C4D excites me so much now after what you have said. I hope you have a wonderful day!

Hello @jenandesign,

Thank you for reaching out to us, I appreciate sharing your customer journey with us. I cannot make any further statements on the ongoing development of Cinema 4D, but we are of course happy to hear that you see value and potential in Cinema 4D.

I will make sure that your input does reach the simulation Team.

Cheers,
Ferdinand