Render Plugin: DirtyState, UniqueId, ActiveDoc

On 12/11/2014 at 05:53, xxxxxxxx wrote:

User Information:
Cinema 4D Version:   15 
Platform:   Windows  ;   
Language(s) :     C++  ;

---------
Hi, me again :-)

This post will be related to uniqueness of objects and information available in this thread:
Thread A: https://plugincafe.maxon.net/topic/8214/10707_uniqueobjectidgetmarkerstampexgetmarker-solved

I'm trying to write some sort of caching system that lets me listen to dirty states of objects (polygon-objects, lights, materials). I'm only interested in objects that have changed compared to my last render call. My procedure works like this:

When a render call is sent, for each object of interest:
1. Create a unique id from the object of interest (refer to thread A on top)
2. Check in a map/container if the object is dirty, if no, go to 3. if yes go to 4.
3. Object is not dirty, so we skip it, CONTINUE with 1.
4. Object is dirty, so we extract valuable information from it
5. Update the dirty state of the object in our map/container, or if it is not present, insert it
6. CONTINUE with 1.

My problem is now, that the unique id and the dirty mechanism of C4D only work if the scene/document stays the same. Unfortunately, C4D copies the document when rendering. This happens, as far as I know, because the artists should be able to keep on working while rendering.

Here I summarize what I have found out about the copying behavior in C4D:
* Editor Render View: My dirty mechanism works great, all unique ids are actually unique, the dirty states are correct.
* Picture Viewer: When rendering a whole animation the whole dirty mechanism fails for the first frame, so everything "has changed". But afterwards my procedure works fine.
* Picture Viewer: When rendering single images, and clicking the Picture Viewer icon one after the other, my mechanism completely breaks. I found out that the dirty states of the object in a test scene stays exactly the same! So, for example, a Cube sitting there might have the dirty state (of all flags for simplicity) of 22. When I move the camera, and hit on the Picture Viewer again, the Cube receives a new unique id from my code, but the dirty state stays the same. So I and C4D know that the object has not changed, but since it is a copy, I cannot get the same id back from the rendering before.
* Interactive Render Region: This is crazy as hell, every time the region is update, all my unique ids change to new ones, so my algorithm breaks.

Now my questions:
* Is it true that all this happens because C4D always copies a document before rendering?
* Is there a way to tell C4D to not copy the document at all? I will promise to return quickly, so that the user won't wait for long!
* Is it somehow possible (and this is actually the same question as in thread A) to get a REALLY UNIQUE OBJECT ID. This should behave like when looking at a scene and saying: "Oh, a green cube up there!" This green cube up there is the same and I can clearly see it, so I want a mechanism in C4D to do exactly this.

I would go back to using the GetActiveDocument() but I was told to not do so, because it can lead to crashes and strange bugs.

In my case the easiest solution would be to stop C4D from copying the document, although I already can hear you all shouting at me! ;-)

Alternatively, please give me an actually unique object id, an id that identifies it. An id that lets me know if it is the same object in the current frame, 100 frames later, stretched, rotated, moved, disabled, etc. In all those cases, it is still the same object and I don't care if C4D wants to copy a scene, I simply want a unique id!

Thanks,
Andreas

On 12/11/2014 at 07:22, xxxxxxxx wrote:

I thought about the same , my solution would be make a tag for objects that needs to be static "so it won't rebuild BVH" , the user would be responsible for static/dynamic object control

On 13/11/2014 at 05:40, xxxxxxxx wrote:

Hello,

In most cases Cinema will create a copy when you want to render the currently active document. One exception is when you call "Render View" (but there is also an exception to this exception). The only situation where you can control this behavior is when you use RenderDoucment(). If you want to check if you are rendering a copy or not check for the RENDERFLAGS_NODOCUMENTCLONE flag.

Please keep in mind that in many situations (Render Queue, Team Render, etc.) there may be no active document or the active document has nothing to do with your render task. In fact you can start rendering the current document in the Picture Viewer and then close the current document while the render process is still running.

If you want to identify an object you can use the unique IP system. You can get the full IP for an given RayObject by GetUniqueID(). This function will return a pointer to an array of Int32 values with the given length. Alternatively you can call GetUniqueIP() on a BaseObject which returns the position in the local hierarchy level.
Generators must take care of setting this IP. This system is intended to allow you to check if a certain object is the same object in one frame and another frame. Because of this Cinema will create the IPs automatically for each external render process (not for rendering in the viewport or using the IRR). You have to check the RENDERFLAGS_EXTERNAL flag to make sure Cinema created the unique IP.

Another thing to use is the unique ID returned by GetGUID(). This value also based on the Marker so it has the same shortcomings.

best wishes,
Sebastian

On 13/11/2014 at 12:04, xxxxxxxx wrote:

Hi,

From the docu I read for RenderDocument() and for its parameter "doc" the following:

The document to render. Can be a clone of the real document to save memory usage. The caller owns the pointed document.

A clone saves memory?! How should a clone do such a thing when it is a copy of another scene?
So do you suggest to invoke a render call on my own if I don't want the scene to be copied? But I want my plugin to behave like the standard renderer of C4D, so all the buffers etc. should be written/returned in the way as C4D wishes them to be. When the usual render call is made, I copy my data back to the bmp that was handed me by C4D. Is it correct, that I don't have a chance to overcome the copying if I want to use the normal render buttons like "Picture Viewer"?

If you want to check if you are rendering a copy or not check for the RENDERFLAGS_NODOCUMENTCLONE flag.

That's find, but what can I do with this information? I only know that my cache system will be broken in this render call but I have not found a way to overcome this^^

About the GetUniqueID() method, see the docu:

Note:  The ID is somewhat constant across frames so it is handy to identify objects in animation.

"Somewhat constant", come on! I cannot rely on a mechanism that already fails in the documentation!

GetUniqueIP() cannot be used with materials, so I would have to use another mechanism for them.

Additionally GetUniqueIP() returns -2147483648 for my polygonobjects or in some cases 0. It's pretty funny that this method simply returns uninitialized memory (at least I guess so) and offers no way to check for failure (e.g. boolean return value, return code parameter, whatever...).
In this case I have to remember that I'm using BaseObjects from the hierarchy discussed in this thread:
https://plugincafe.maxon.net/topic/8226/10722_relations-baseobjectrayobjectopolygon-solved

So I have a RayObject (only O_POLYGON exist, because I turned off 'render perfect' of all parametric spheres first) use the .link to get its corresponding BaseObject and work on this one with my caching system. As you can find in the thread there is additionally magic involved to get the dirty mechanism to work by using GetCacheParent() to the "real" BaseObject. This is all a huge mess and I'm not sure how one can create such an awful object hierarchy in the API... but fine. Maybe I have to repost the code for my hierarchy and my dirty mechanism?! So in the end I get 0 as Id for some of them and uninitialized memory for others.

Sebastian please refer to this thread:
https://plugincafe.maxon.net/topic/8214/10707_uniqueobjectidgetmarkerstampexgetmarker-solved&KW=GetUniqueIP&PID=41748#41748

here we talked about all different ways of identifying objects in C4D and the result was my code at the end. So is this wrong? Do you know a better way?
[EDIT:] you might also want to refer to this thread regarding failed attempts of creating unique ids:
https://plugincafe.maxon.net/topic/7582/9505_getguid-fails-when-rendering&KW=GetUniqueIP&PID=37514#37514

I can think of one way to handle all this:
* User hits "Render View", "Picture Viewer", etc. and I listen to this event.
* When I receive this event I use the current/active document and call RenderDocuement() on my own, render everything and prepare the buffers for C4D. While this happens the application "freezes" because I'm doing my stuff, so no window change etc. can happen.
* Then the "real" render code comes in, but I'm already finished and only have to copy the buffers where C4D likes them to be. If the document changes in the background or anything else bad happens, I don't care anymore, because I'm already finished.

This sound like a crazy and user-unfriendly hack ;-)

Thanks,
Andreas

On 14/11/2014 at 08:57, xxxxxxxx wrote:

Hi,
I agree with you, that the proposed hack will not lead to a smooth user experience.
As Sebastian already elaborated the different existing mechanisms to identify objects and none of these seems to fulfill your needs, I'd like to do a step back for a moment.
Please be patient with me, I only have a very faint idea, how your project looks like and what you are trying to achieve.
You already said, you are doing your own renderer. And in my imagination you are trying to optimize it in a way, that you only need to transfer/transform changed data (be it geometry, be it materials, textures) to your renderer. While I understand your concern, you should take into account that C4D is copying the scene in certain cases for good reasons. And I think, you should consider, if it wouldn't be better to rebuild your entire data from the given copy of the scene, rather than trying to detect changes in these cases.

Now I can already see you jumping up and down, calling me names. But hope we can discuss this peacefully :wink:

On 14/11/2014 at 10:06, xxxxxxxx wrote:

there is a solution "not sure if it is (correct)" but it may work, use SetUniqueIP() , at the first render, you just set each object from your RayObject(s) , after that, when you render again, check all ray objects that are set "got a valid unique ip in a known range that you define" , and if the object is dirty, send it, if it has invalid unique ip, give it a new unique ip, and send it, this approach will let you control the IPs, but I sense it is pretty dangerous

On 14/11/2014 at 10:09, xxxxxxxx wrote:

I'd rather recommend to use a hidden tag for this.

On 15/11/2014 at 04:02, xxxxxxxx wrote:

I'd rather recommend to use a hidden tag for this.

I think this might be the right track! :-)

I will try the following:
* Create a tag that simply stores a unique id in its container, the ids are, for example, created by simple incrementing a windows HUGE ^^
* When a render call happens, iterate over all objects in the scene and attach the tag to them, if they don't already have the tag
* Now if I want to identify an object, I can simply check for the tag because the id in the tag will not change even when the scene is copied.
* In the end I will hide the tag (I guess that this is easy?!)

I report back, if I have tested this approach.

Thanks Andreas for this hint! :-)

On 15/11/2014 at 04:05, xxxxxxxx wrote:

To be honest, the idea came from Sebastian, when I discussed this topic with him. Nevertheless glad it may help.

On 17/11/2014 at 10:09, xxxxxxxx wrote:

Hi all,

I don't think that storing a unique ID in a hidden tag would solve all the problems. Think of generators that clone objects, i.e. emitters, arrays mograph and so on.

You will end up with multiple objects with the same unique ID.

That might be helpful in a few cases, but when you want to keep track of objects across several frames (for motion blur), it won't work.

I would also like to see some way to actually be able to uniquely identify objects *reliably*. None of the available methods actually work.

* Markers: will not give consistent unique IDs.
* Unique IPs: only work properly in external renderer, somewhat consistently, but I have a scene or two where they fail. Magic number IPs for generators and black magic IP generation that only works in the external renderer? Has to be some sort of inside joke :)
* GetGUID: doesn't give consistent IDs and fails in the same cases as the unique IPs. It's based on them, not sure what I expected.

But even the best IP system doesn't help you if you want to track changes in a scene, as it will break when changing the order, removing or inserting. (In this case the hidden tags might help).

On 17/11/2014 at 12:25, xxxxxxxx wrote:

Welcome fused to this old unanswered topic^^

I have thought about the tag system and there are a lot of problems related to copy pasting objects from one scene to another and there are also problems related to undo/redo functionality.

The interesting part is, that C4D has to keep track of objects inside it too. Is it possible to hear a comment of someone actually developing C4D?

On 17/11/2014 at 12:36, xxxxxxxx wrote:

Oh and another one:
Is it possible to identify a document? Let's say my open scene is named "TestScene.c4d". When I render into the Render View the document should be the same, so no copy. I can check this with RENDERFLAGS_NODOCUMENTCLONE but I want to identify a copy as well. So, for example, my original TestScene.c4d has id 12345 and when I work on a copy the id of the document is, e.g. 987768. This is important because I have found that my caching system even breaks when materials should be checked for dirty states because they seem to be a copy too. So I need to setup a separate caching system for each document C4D copies/creates on the way down the rendering pipeline.

Maybe a C4D programmer can join this discussion here?

On 17/11/2014 at 15:29, xxxxxxxx wrote:

@FrozenTarzan I think this is an unavoidable behavior!!
in Maxwell render, and even in Cinema 4D standard and physical render, once you click render it will start creating BVH from scratch every time

On 18/11/2014 at 19:27, xxxxxxxx wrote:

For your information FrozenTarzan, Sebastian is currently working in C4D's rendering code. You can't get closer to a C4D developer than this.

The workflow in C4D is to clone the scene sent for renderering. As MohamedSakr pointed out, All professionnal renderer plugins live fine with this. I suggest to either go with the flow or follow one of the solution ideas discussed.  The tag with an ID seems promising to me and yes there will be complexity to resolve around generators and procedural objects.

Unless you have questions about our current APIs I'm not sure we will be able to help you much.
best regards,

Jean-Francois Yelle
Development Support Manager
MAXON GmbH.

On 19/11/2014 at 04:08, xxxxxxxx wrote:

Hi Jean-Francois Yelle,

I'm trying hard to be polite, but your answer has nothing to do with my questions. You don't have to sell your API here, you don't have to convince me that your API in C4D is well designed. I'm already developing with it. And yes my project might be good advertisement for Cinema4D as a whole in the future. All I'm asking is to receive honest and clear answers to simple questions. It is absolutely ok to tell me "Yes every engine, every software handling objects in any way has to have a solution for uniquely identifying them. Of course C4D does this as well internally, but our API does not offer a consistent, unique and stable way to do this from a user/programmer/developer perspective." Non, absolutely not a single one of the ideas presented in this forum is the solution to my problem. It is ok to admit this.

And again, there is no problem with admitting problems in C4D, it is no problem to tell us that you don't know the answer. It is also ok to admit that C4D's API is old, really old and based on strange C-style patterns. This is the normal way a software grows when time goes on. But I beg you to give honest and clear answers to questions. Sebastian, for example, have posted that there seems to be a bug in the render region functionality here: https://plugincafe.maxon.net/topic/8204/10691_videopostdata-plugin-render-region-not-working
This is so cool, because since then I know that it is not my fault and I can go on with the project, so thanks again! :-)

I have a team behind me for this project and we will go on working hard on it, so I'm looking forward to interesting and productive discussions here in the plugincafe community.

Thanks,
Andreas

On 19/11/2014 at 06:47, xxxxxxxx wrote:

Andreas,

C4D is 20 years old, and has a rendering pipeline built around creating a copy of the scene which enables the user to his work while rendering.
In this rendering pipeline, our internal UID system cannot be used to recognize scene changes and relate to the cloned object. Sebastian's proposal to use your own UID in tags around a variation of a subject/observer design pattern still seems to us like your best option.

Sebastian and Andreas are rock solid, but unless you explain them what you are really trying to achieve, and that we find something better to suggest than what has already been said, we will have to apologize and say our API support job is done. We don't have a solution to everything unfortunately.

If you need to discuss more about your development issue, you should provide more details on what you really are trying to achieve. Right now, we're all out of ideas to help you out.
If you need to discuss on how to have a productive conversation with me or my team or to talk about consulting services, I invite you to mail me at [email protected].

Jean-Francois Yelle
Development Support Manager
MAXON GmbH.