Here are my results after 35 minutes of processing:
Much better than the 12 hours and 0 results I got from my previous attempt 
It looks like there are 5,183,087 segments. I have them split across 519 spline objects. The scene's viewport navigation is still fairly responsive, about 25fps when all the splines are visible. Higher if I zoom into a section.
I'd still like to improve the method if anyone can provide more information on how I might use SetAllPoints for the splines here.
@zipit said in Generating Splines from JSON Data:
- Aside from instantiating the
SplineObjectin the first place, you would also have the problem that Cinema's splines are not static, i.e. are being dynamically cached. That would mean all this point data had to be reprocessed each time the cache for this spline is being build.
Perhaps this is a reason it would be good to build splines made of fewer segments/points? I will not be modifying the curves after they're built the first time. Are you saying that the cache is built for the SplineObject even outside of the call to SplineObject.Message(c4d.MSG_UPDATE)? That's not the way it seems to behave based on my (limited) observations.
- This is more an academic point due to the mentioned problems, but setting each point individually seems very inefficient, you should push all points in at once, using
PointObject.SetAllPoints.
I'd like to use this method but I haven't found any examples of using it on a SplineObject. My concern is that I would have to call this once per SplineObject (as opposed to per segment) which would entail keeping a 10,000 element python list of which each entry is a 5 element python list of c4d.Vector alive until the SplineObject points are ready to be set. I might be able to use one of the methods from itertools to do this relatively quickly, but I'm just not sure if it'll actually be an improvement. I suppose it would be if the cache is in fact being built whenever a point is added.
- The only way to go would be to reduce the dataset in dimensionality, which would mean for a curve to do some curve fitting. Cinema has a curve fitting function in its API, its even accessible in Python in the
c4d.utilsmodule, but I somehow doubt that it is up to the task. You will probably have to usenumpyandscipyfor that.
Each curve is already very simple; just 5 points that Disney expects to be interpolated (I'm using B-Spline in this case). Perhaps lowering the number of intermediate points for this case would be beneficial, especially as these splines are very distant from the majority of the scene (occupy a small section of the frame).
I've gone ahead and modified my code to chunk the curve data so that each SplineObject ends up with only 10,000 segments. Now processing 100,000 curves takes only 42s down from the prior 10m13s. This seems to be a roughly linear increase in time from the 5s for 10,000 segments that I'd previously recorded... so I think I'm happy for now?
point_count = 0
segment_count = 0
spline_count = 1
num = 10000
spline = srcSpline.GetClone()
spline.SetName("{0}_{1:0>4d}".format(name, spline_count))
curves = ijson.items(json_file, 'item', buf_size=32*1024)
for i, curve in enumerate(curves):
index = i % num
if index >= (num - 1):
spline.InsertTag(texture_tag.GetClone())
spline.InsertTag(object_tag.GetClone())
spline.SetLayerObject(layer)
spline.Message(c4d.MSG_UPDATE)
doc.InsertObject(spline, group)
point_count = 0
segment_count = 0
spline_count += 1
spline = srcSpline.GetClone()
spline.SetName("{0}_{1:0>4d}".format(name, spline_count))
if spline_count > 10:
break
point_count += len(curve)
segment_count += 1
spline.ResizeObject(point_count, segment_count)
for id, point in enumerate(reversed(curve)):
spline.SetSegment(segment_count-1, len(curve), False)
spline.SetPoint(point_count-id-1, flipMat * c4d.Vector(*(float(p) for p in point)))
Hello!
I'm revisiting the Moana Island data set and I'm making great progress; I've got almost all of the assets converted into Redshift Proxies.
The biggest problem I'm currently facing is a 3GB JSON file that defines renderable curves on the largest mountain asset. I don't know exactly how many curves are defined in this file, but based on the curve count and data size of other JSON files I think its roughly 5.2 million curves. Each point of the curve is an array with 3 items, each curve is an array of N points, and the curves are stored inside of a top level array.
The built-in json module must load the entire file into memory before operating on it. I've experienced extremely poor behavior on any JSON file over 500MB with the json module so I am instead parsing the files with ijson which allows for iterative reading of the JSON files as well as a much faster C backend based on YAJL.
Using ijson I was able to read an 11GB file that stored transform matrices for instanced assets on the beach. However, even using ijson I cannot seem to build a spline from the curves in this 3GB file (I gave up after letting the script run for 12 hours). I have a suspicion it has more to do with the way I'm building the spline object than parsing the data. So I have some questions. Is there a performance penalty for building a single spline with millions of segments? Should I instead build millions of splines with a single segment? Or would it be better to try and split the difference and build thousands of splines with 10,000 segments each?
I've done a little performance testing with my current code and right now it takes 10 minutes 13 seconds to build a single spline out of the first 100,000 curves in the file. However, if I build just the first 10,000 curves it only takes 5 seconds.
I'm leaning heavily toward chunking the splines into 10,000 segment batches but I want to first see if my code could be further optimized, here is the relevant portion:
curves = ijson.items(json_file, 'item', buf_size=32*1024)
#curves is a generator object that returns the points for each segment successively
for i, curve in enumerate(curves):
#for performance testing I'm limiting the number of segments parsed and created
if i > num:
break
point_count += len(curve) #tracking the total number of points
segment_count += 1 #tracking the number of segments
spline.ResizeObject(point_count, segment_count) #resizing the spline
for id, point in enumerate(reversed(curve)):
spline.SetSegment(segment_count-1, len(curve), False)
spline.SetPoint(point_count-id-1, flipMat * c4d.Vector(*(float(p) for p in point)))
spline.Message(c4d.MSG_UPDATE)
I never would've found the command id so thanks a ton for sharing that. Marking as solved.
Well, it was a beautiful dream. The CallUVCommand to UVCOMMAND_AUTOMATICUV still fails when supplying a settings container based on the MDATA I found. In retrospect it does seem odd that these would be MDATA when the other UV command settings are not.
Looking forward to your examples. I'll keep stabbing blindly in the dark in the meantime.
@m_adam Thanks Maxime!
I'd found UVCOMMAND_AUTOMATICUV while perusing the depths of symbols.h yesterday but I couldn't find a way to make it work and I wasn't sure if it was what I needed... so I stopped looking into it. Your post renewed my enthusiasm and I went looking in the C++ SDK docs to find this:
Which I believe will allow me to create my settings container for the command. I'll try this later today and let you know the results.
It would be nice to have the python documentation updated too 
@wuzelwazel said in Packed Automatic UVs Command in S22?:
Alternatively being able to invoke the Set UVW From Projection... command with the automatic packed option would be even more straightforward in this case. Is that possible?
I should mention that I can use c4d.CallCommand(c4d.UV_SET_FROM_PROJECTION_CMD_ID) but in that case I don't know how to set the options for the command via Python.
Hello, I'm struggling to figure out how to call the new Packed Automatic UV command from Python. I've found the BaseContainer where I can modify settings, but haven't figured out how to invoke the command.
Alternatively being able to invoke the Set UVW From Projection... command with the automatic packed option would be even more straightforward in this case. Is that possible?
@kbar said in Ptex Shader Progress:
I had planned on doing this again and converting the OBJ files to C4D files in the process, to make them smaller and faster to load. Which is what you are doing with Redshift proxies I believe?
Yes, the Redshift Proxies work as a sort of reference so the main c4d file would be quite small. In addition the display of each proxy can be simplified to either reduce or eliminate the viewport being bogged down with drawing faces. Of course, when everything is just a bounding box the scene doesn't look like much before rendering!
The other benefit of the proxy is specific to Redshift in that the mesh will not need to be processed before the BVH is constructed and rendering can begin.
@kbar I have seen your importer! I even tried to contact you through your website's form to share the python script that I'd been working on to convert the dataset in the hopes it would be helpful. I'd solved correction of the transform matrices and had placement of instances, archives and curves working really well.
My goal was a bit different though, I wanted to convert the Ptex textures into UDIMs, build the materials, build Redshift proxies from the various assets and then instance them into the scene. There were only 2 things that held me back at the time:
OpenImageIO couldn't read many of the Ptex files properly.
The enormous size of some of the JSON files made accessing them with the built-in Python json module incredibly difficult as it would attempt to load the entire file into memory.
What I'm trying to do now is solve #1 by circumventing OIIO all together. Although after posting this shader I realized that it would be much more efficient to create a command plugin I could call from Python that would:
-Read the associated Ptex files
-Setup the UDIM UVs on the objects
-Write the Ptex sub-images directly into a bitmap following the UDIM layout
That wouldn't be helpful in a general Ptex workflow, but would get me what I need much more efficiently (no need to bake textures in a separate pass).
As for #2 I did find the ijson module, which would allow me to iterate through the larger json files without loading them entirely into memory.