Solved Avoiding invalidating saved scenes when implementing Read/Write

Hello there!

I'm not quite sure if the issue I'm having is already explained elsewhere, although I did try to find guides and articles on this, but found nothing sufficient enough.

So I implemented a SceneHook which writes data when a document is written and reads data when a document loads. That's fine and not the issue I'm having.

The data that this SceneHook handles cannot be put into a container, but also has no effect on the contents of the scene. When a scene is now saved with that data internal to this SceneHook, it cannot be opened anymore without the plugin being installed, as Cinema reports an 'incorrect file structure' otherwise, basically corrupting every scene saved while this plugin is installed.

I was under the impression that Cinema simply ignores data for nodes from a missing plugin, so I guess my implementation of the Write() function is just not quite right.

Is there any documentation as to how reading and writing should be implemented, or is that actually expected behaviour?

@doppelgamer said in Avoiding invalidating saved scenes when implementing Read/Write:

The data that this SceneHook handles cannot be put into a container,

Why?

I guess my implementation of the Write() function is just not quite right.

I think it would make sense if you would show us your code here.

Read()/Write() operates on a HyperFile. You find some examples in the HyperFile Manual and NodeData::Read() / NodeData::Write() Manual

@PluginStudent said in Avoiding invalidating saved scenes when implementing Read/Write:

Why?

Because the SceneHook makes use of plug-in internal classes. I guess it technically is possible to put this data into the node's container, however that would be far less efficient.

I think it would make sense if you would show us your code here.

I would if I was sure that it's unexpected behaviour by Cinema to not be able to read files anymore after a plug-in manually wrote into the file. The code is fairly standard abstraction of classes into primitive types, using the HyperFile's read/write interfaces.

Read()/Write() operates on a HyperFile. You find some examples in the HyperFile Manual and NodeData::Read() / NodeData::Write() Manual

Thank you for the links! Even though I read them carefully, I couldn't find anything I did any different, besides code within the Read() function checking for the 'value header', which mine doesn't.

However, reading isn't the issue I'm having, or rather reading for my plug-in within my code. The problem is that scenes saved while this plug-in is installed cannot be opened anymore when the plug-in is not installed.

Is this expected behaviour, and would putting this data into the node's BaseContainer solve this issue?

Hi,

no, normally a missing plugin should not trip Cinema on loading a file where that plugin has had serialised some data into that file via NodeData::Write, as it would be just a dormant HyperFile block then.

However, I am not quite sure about SceneHooks and if they are even supposed to be written into the document. It could very well be that the usual if I don't know it, I put a dummy node in-logic does not apply to a SceneHook. I always viewed a SceneHook as, well a hook, so something that should not be serialised, as it is unique to each context it runs in. For the same reason I would also doubt that the data container of a SceneHook is being serialised per document, but I might be wrong about this.

Cheers,
zipit

@zipit said in Avoiding invalidating saved scenes when implementing Read/Write:

no, normally a missing plugin should not trip Cinema on loading a file where that plugin has had serialised some data into that file via NodeData::Write, as it would be just a dormant HyperFile block then.

Yeah that's what I experienced with for example a BaseObject.

However, I am not quite sure about SceneHooks and if they are even supposed to be written into the document. It could very well be that the usual if I don't know it, I put a dummy node in-logic does not apply to a SceneHook. I always viewed a SceneHook as, well a hook, so something that should not be serialised, as it is unique to each context it runs in. For the same reason I would also doubt that the data container of a SceneHook is being serialised per document, but I might be wrong about this.

Now that I think about it, you may be right, a SceneHook seems a little unsuitable for what I'm trying to do. It is doing something on its own which it uses scene-specific data for, which is what I'm writing to and reading from the Hyperfile.

Is there a more suitable way to store scene-specific data within a BaseDocument that is a little too complex to be put into its BaseContainer? I guess utilising GeData, although that could be a little overkill.

I still don't get what data would be too complex to write into a BaseContainer.

You can store raw memory in a BaseContainer.

If you only want to read/write to the document's BaseContainer, you could listen to the MSG_DOCUMENTINFO sent when the document is loaded and saved.

@PluginStudent said in Avoiding invalidating saved scenes when implementing Read/Write:

I still don't get what data would be too complex to write into a BaseContainer.

'Too complex' might be the wrong wording, it's more like the data is laid-out into classes with relations within each other. Again, storing this data within the NodeData's container is technically possible, but quite inefficient.

You can store raw memory in a BaseContainer.
If you only want to read/write to the document's BaseContainer, you could listen to the MSG_DOCUMENTINFO sent when the document is loaded and saved.

I guess that's going to be a mixture I'm going to utilise. Thank you!

Maybe someone from Maxon could look into this issue, I think it's quite important to communicate to plug-in developers that making a SceneHook write into a HyperFile will invalidate the scene file for future use without the respective plug-in.

hi,

creating a scenhook or any kind of plugin will not make the hyperfile structure invalid. A SceneHookData inherit from NodeData.
It does check if the plugin exist or not before calling the Read function.

I couldn't reproduce the file structure error you are talking about.
Maybe because my data structure is too simple (just a Int32)
Can you try with a simplified data structure ? Share a simplified version of your code ? you can use [email protected]

Cheers,
Manuel

@m_magalhaes Interesting, I just tried that myself, writing/reading only an Int32 does not invalidate the file, even after the plug-in is removed. It seems like my previous suspicion was right and it's my code that is invalidating the structure.

Alright, this is my (somewhat modified) code which writes into the HyperFile:

Bool MySceneHookData::Write(GeListNode * node, HyperFile * hf)
{
    if (!hf->WriteUInt32(UInt32(_descriptors.GetCount())))
        return false;
    
    for (auto entry = _descriptors.Begin(); entry != _descriptors.End(); ++entry)
    {
        auto & desc = entry->GetValue();
        
        if (
            !hf->WriteUuid(entry->GetKey()) ||
            !hf->WriteString(desc.name) ||
            !hf->WriteTime(desc.minTime) ||
            !hf->WriteTime(desc.maxTime)
            )
            return false;
    }
    
    if (!hf->WriteUuid(_activeKey))
        return false;
    
    return SceneHookData::Write(node, hf);
}

In my code, _descriptors is an instance of maxon::HashMap<C4DUuid, Descriptor> where Descriptor is a struct laid out basically as written in the Write function.

Now after reading the articles linked by @PluginStudent I realised that I should make use of chunks when writing data collections. I didn't come around to change that yet, though I definitely will do that now that I know the code should work.

hi,

I still cannot reproduce your issue.

Some remark about your code, you can use maxon::Uuid instead of C4DUuid look at that manual for more information.
You don't need to return SUPER::Write you can simply return true or false.

and yes @PluginStudent is almost always right ^^ (and almost is just to not say always :p)

Cheers,
Manuel

@m_magalhaes said in Avoiding invalidating saved scenes when implementing Read/Write:

Some remark about your code, you can use maxon::Uuid instead of C4DUuid look at that manual for more information.

I guess it would've been helpful to mention that earlier, but I'm "stuck" at R16 and as far as I know, that's not yet implemented in this version.

You don't need to return SUPER::Write you can simply return true or false.

Oh well that's actually something I "corrected" after I began this thread. I saw that in an example for HyperFile handling so I just repeated it.

I still cannot reproduce your issue.

Well that's weird. I created a new document within Cinema, only added a single Sphere object, and even without any data within my SceneHook I can reproduce this issue. So essentially, my code executes two things:

hf->WriteUInt32(0);
/* code that won't be executed anyway */
hf->WriteUuid(C4DUuid());

Which seems to be enough to invalidate the scene file without the plug-in installed.

Lo and behold! What seems to be the problem (in my case, anyway) is writing the C4DUuid last, because writing just the UInt32 doesn't cause any problem.

EDIT: fixed some words/typos

@doppelgamer said in Avoiding invalidating saved scenes when implementing Read/Write:

hf->WriteUuid(C4DUuid());

ok, this is definitely causing issue. If you just try to open the file you just saved, you have invalid file structure.

I've opened a bug report entry for that one and send an email to our devs.

Cheers,
Manuel

@m_magalhaes Nice!

As I understand, any call of WriteUuid on the HyperFile will cause this issue.

Just my luck to stumble on an unknown bug. 😅

Thanks for your support, @m_magalhaes !

hi,

well not "any" but this one does. I'm not even sure if it's the call itself that does it.
If you do that in your own HyperFile for example, there's no problem.

Cheers,
Manuel

So as I'm most likely going to not be able to make use of this fix when it rolls out, I had to resort to "hack" the C4DUuid into the HyperFile.

Writing:

C4DUuid uuid;
hf->WriteMemory(&uuid, sizeof(uuid))

Reading:

bool read_uuid_hack(HyperFile * hf, C4DUuid * uuid)
{
    Int size;
    void * data = nullptr;
    
    if (!hf->ReadMemory(&data, &size))
        return false;
    
    bool matched = (size == sizeof(*uuid));
    
    if (matched && uuid != nullptr)
        *uuid = *reinterpret_cast<C4DUuid *>(data);
    
    DeleteMem(data);
    
    return matched;
}

Using this "hack" avoids invalidating the file.

I'm aware of the risks that come with this and I would generally strongly advise against it, but since these UUIDs are only used to identify items internally within my plug-in it won't do any harm.

hi,

or just use GetString to retrieve Uiid in a string form and CopyFrom to create your c4duuid.

But as you are stuck in R16 i didn't test it. (CopyFrom is returning a maxon::Result<void> now)

Cheers,
Manuel

@m_magalhaes Ah you're right, for some reason I overlooked those methods.

So for writing:

hf->WriteString(uuid.GetString());

And for reading:

C4DUuid uuid;
String uuidString;

hf->ReadString(&uuidString);
uuid.CopyFrom(uuidString);

Much cleaner and actually supported by Cinema.

Very nice, that solved my issue in a satisfactory way! 🙂

hi,

back to the subject, i was tracking the bug and I discovered it was on my chair. After messing around too much about the code, I was doing something very wrong.

when you register your scenehook what is the level ? (last parameter of RegisterSceneHookPlugin)

Cheers,
Manuel