3D RGBD Camera & C4D Import



  • On 27/09/2017 at 02:59, xxxxxxxx wrote:

    User Information:
    Cinema 4D Version:   r18/19 
    Platform:   Windows  ; Mac  ;  Mac OSX  ; 
    Language(s) :     C++  ;  XPRESSO  ;

    ---------
    Hi

    Paul Melvin at Maxon recommended I get in touch with regards to some API, SDK and C++ work for an exiting new project of mine that is underway and one that is set to be very experimental in terms of 3D scanning and positional tracking.

    The project is a music video for Avalon Emerson, an emerging female techno producer that is gaining publicity quickly all around the world.

    We're shooting elements of the project at Kew Gardens in London with Stereolabs' RGBD "Zed" camera. We'll use the data gathered from it to match-move our 4K camera and extract 3D elements of the scenery for compositing. The cool thing about the Zed is that it uses stereo odometry and depth sensing to generate textured meshes which can be used for all kinds of applications from robotics, drones, 3D printing and most excitingly for me, VFX!

    > > Right now I'm facing a few complications that will require help using C++ with the Zed's SDK. The main problem I'm facing is the Zed's internal coordinate system. I need to export Zed's quaternion XYZ positional/rotational data to Cinema 4D's HPB system and struggling to get my head around it all. I'm not a C++ developer, but Stereolabs have assured me it is a simple task. With the shoot date set for the 7th and 8th of October, and the hand-in for the 17th, I'd love to have the workflow ready.

    > >
    >
    For the Zed's SVO files, I'll ideally need a built exporter to extract the camera's positional data and textured OBJ mesh ready to open in C4D without the need for any additional calibrations to the rotational order and PSR. I'm thinking the ultimate format would be FBX as I believe it can hold camera and mesh data and is readable in other 3D apps. There would need to be some attention to frame-rate too as the recommended recording settings for the ZED are 60fps at HD720.

    >
    >
    There are no sample SVO files available for download as far as I know but my teammate has shot some material with the Zed which he may be able to forward over to you when he gets a moment.

    >
    >
    Here's an overview of the coordinates system:

    > Zed Positional Tracking

    Here are the more in-depth links for Zed's SDK and API:
    SDK Download

    > Zed Documentation

    > Zed API Referance

    > Zed Classes, Structs, Unions and Interfaces

    For now, I've attached some already-extracted elements from the Zed's SVO recording format.
    https://drive.google.com/open?id=0Bw4AQ_iwgwBpbVFuWlhFQXpfUzA
    https://drive.google.com/open?id=0Bw4AQ_iwgwBpVlRld1VJNnBwcFk
    https://drive.google.com/open?id=0Bw4AQ_iwgwBpZjNZcnpDM0E4YTg

    We've been looking for solutions all over and haven't managed to find anything particularly user friendly yet so It would be great if you could let me know your thoughts with regards to this all!

    Kind regards,

    Hayden Martin
    07817281488



  • On 28/09/2017 at 02:28, xxxxxxxx wrote:

    Hi Martin, first of all welcome to our Cinema 4D development-related community.

    With regard to your support request, although your request looks complete in terms of provided documentation, we , as the SDK Team, can't develop plugins for our customers. That said, usually our developers community has proved to be supportive on this kind of requests usually getting privately in touch with member asking for developing a plugin.

    So far, I encourage you to have a look at our Matrix and Quaternion classes reference and to the Matrix Manual where the topic you're looking for is properly covered. On top of this, looking at the data provided, I actually see no quaternion-formatted data in the .CSV but simply XYZ-rot and XYZ-pos values which should be easily loaded in Cinema 4D.
    To me a simple script, parsing your .csv line-by-line and setting the camera movement at specified frame, could easily make your day. Finally, what still puzzle me is the TimeStamp (in nanoseconds) which actually never change from the beginning to the end of the file.

    Best, Riccardo



  • On 28/09/2017 at 04:15, xxxxxxxx wrote:

    Here a sample for doing what you want to achieve in python.
    There is small issue regarding coordinate and orientation since it's appear to not be the one describe here http://www.stereolabs.com/documentation/overview/positional-tracking/coordinate-frames.html

    So maybe a picture with the actual world axis and the mesh + the track of the camera would be nice.

    Moreover I'm not sure, I still need to investigate but I guess my script suffer of a memory leak in AddKey function. Then use it with care.

    #Import csv stereolabs inside c4d
    # https://plugincafe.maxon.net/topic/659/13815_3d-rgbd-camera--c4d-import
    # graphos 28.09.2017
    # THIS VERSION ACTUALLY NOT WORK FULLY => DO NOT USE IT IN PRODUCTION
    # USAGE:
    #   Copy - past in Script manager
    #   Select a camera
    #   Run
    #   Select csv file
    #   Enjoy ! :)
    import c4d
    import csv
      
      
    class Data(object) :
        timestamp = None
        rot_x = None
        rot_y = None
        rot_z = None
        pos_x = None
        pos_y = None
        pos_z = None
      
        def __init__(self, data) :
            if not isinstance(data, str) : return
      
            # Build a list from string
            datas = data.replace(" ", "").split(";")
            buffer = list()
            for i, value in enumerate(datas) :
                try:
                    x = None
                    if not i:
                        x = int(value)
                    else:
                        x = float(value)
                    buffer.append(x)
                except:
                    pass
      
            # Check list is correct
            if len(buffer) != 7: return
      
            # Assign value
            self.timestamp = buffer[0]
            self.rot_x = buffer[1]
            self.rot_y = buffer[2]
            self.rot_z = buffer[3]
            self.pos_x = buffer[4]
            self.pos_y = buffer[5]
            self.pos_z = buffer[6]
      
        def __str__(self) :
            return "time: {0}, rot_x: {1}, rot_y: {2}, rot_z: {3}, pos_x: {4}, pos_y: {5}, pos_z: {6}".format(
                self.timestamp,
                self.rot_x,
                self.rot_y,
                self.rot_z,
                self.pos_x,
                self.pos_y,
                self.pos_z
            )
        
        def _getChildrenWorld(self, obj) :
            buffer = list()
            children = obj.GetChildren()
            for child in children:
                buffer.append(child.GetMg())
      
            return buffer
      
        def _setChildrenWorld(self, obj, list_matrice) :
            children = obj.GetChildren()
            if len(children) != len(list_matrice) : return False
      
            for i, matrice in enumerate(list_matrice) :
                doc.AddUndo(c4d.UNDOTYPE_CHANGE, children[i])
                children[i].SetMg(matrice)
      
            return True
      
        def setWorldPosition(self, obj) :
            m = obj.GetMg()
            m.off = c4d.Vector(-self.pos_x, self.pos_y, self.pos_z)
      
            doc.AddUndo(c4d.UNDOTYPE_CHANGE, obj)
            obj.SetMg(m)
      
        def setWorldRotation(self, obj) :
            old_pos = self._getChildrenWorld(obj)
      
            doc.AddUndo(c4d.UNDOTYPE_CHANGE, obj)
            rot = c4d.Vector(-self.rot_x, self.rot_y, self.rot_z)
            m = c4d.utils.HPBToMatrix(rot)
      
            m.off = obj.GetMg().off
            obj.SetMg(m)
      
            self._setChildrenWorld(obj, old_pos)
      
      
    def checkValidCSV(filepath) :
        if filepath[-3:] != "csv": return False
      
        file = open(filepath, "rb")
        try:
            line = file.readline()[:-2]
            str_to_check = "Timestamp(ns);Rotation_X(rad);Rotation_Y(rad);Rotation_Z(rad);Position_X(m);Position_Y(m);Position_Z(m);"
        finally:
            file.close()
      
        if line != str_to_check: return False
      
        return True
      
      
    def getTrack(obj, trackID, channelID) :
        # Find the Track
        param = c4d.DescID(c4d.DescLevel(trackID, c4d.DTYPE_VECTOR, 0),
                           c4d.DescLevel(channelID, c4d.DTYPE_REAL, 0)
                           )
        track = obj.FindCTrack(param)
      
        # Create if no track found
        if not track:
            track = c4d.CTrack(obj, param)
            doc.AddUndo(c4d.UNDOTYPE_NEW, track)
            obj.InsertTrackSorted(track)
      
        return track
      
      
    def addKey(obj, track, frame, value) :
        # init var
        time = c4d.BaseTime(frame, doc.GetFps())
        curve = track.GetCurve()
      
        # Find the key
        key = curve.FindKey(time)
      
        found = False
        # If there is no key create one
        if not key:
            # init our key
            key = c4d.CKey()
            track.FillKey(doc, obj, key)
      
            # Set value for our key
            key.SetTime(curve, time)
            key.SetValue(curve, value)
      
            # Insert our key
            doc.AddUndo(c4d.UNDOTYPE_NEW, key)
            curve.InsertKey(key)
      
        # If we found an already existant at current time, jsut set the the value.
        else:
            key = key["key"]
      
            # Set new key value
            doc.AddUndo(c4d.UNDOTYPE_CHANGE, key)
            key.SetValue(curve, value)
      
      
    def main() :
        # Get selected camera
        obj = doc.GetActiveObject()
        if not obj or not obj.CheckType(c4d.Ocamera) : return
      
        # Open file and check validity
        filepath = c4d.storage.LoadDialog(c4d.FILESELECTTYPE_ANYTHING, "Select csv data")
        if not filepath: return
        if not checkValidCSV(filepath) : return
      
        # Read data and store it in list(datas)
        datas = list()
        with open(filepath, "rb") as file:
            reader = csv.reader(file)
            next(reader)
      
            for row in reader:
                datas.append(Data(row[0]))
      
        # Create Keyframe for each data
        doc.StartUndo()
      
        # I acutally limit to the first 600 frame since there is memory leak and we can already so rotation are not good
        for i, data in enumerate(datas[:600]) :
            # Set Frame
            doc.SetTime(c4d.BaseTime(i, doc.GetFps()))
            c4d.DrawViews(c4d.DRAWFLAGS_FORCEFULLREDRAW)
            c4d.GeSyncMessage(c4d.EVMSG_TIMECHANGED)
      
            # Get tracks
            posTrackX = getTrack(obj, c4d.ID_BASEOBJECT_POSITION, c4d.VECTOR_X)
            posTrackY = getTrack(obj, c4d.ID_BASEOBJECT_POSITION, c4d.VECTOR_Y)
            posTrackZ = getTrack(obj, c4d.ID_BASEOBJECT_POSITION, c4d.VECTOR_Z)
            rotTrackX = getTrack(obj, c4d.ID_BASEOBJECT_ROTATION, c4d.VECTOR_X)
            rotTrackY = getTrack(obj, c4d.ID_BASEOBJECT_ROTATION, c4d.VECTOR_Y)
            rotTrackZ = getTrack(obj, c4d.ID_BASEOBJECT_ROTATION, c4d.VECTOR_Z)
      
            # Get world data
            current_frame_world_pos = data.setWorldPosition(obj)
            current_frame_world_rot = data.setWorldRotation(obj)
      
            # Get relative data
            current_frame_local_pos = obj.GetRelPos()
            current_frame_local_rot = obj.GetRelRot()
      
            # Add Key for current value
            current_frame = doc.GetTime().GetFrame(doc.GetFps())
            addKey(obj, posTrackX, current_frame, current_frame_local_pos.x)
            addKey(obj, posTrackY, current_frame, current_frame_local_pos.y)
            addKey(obj, posTrackZ, current_frame, current_frame_local_pos.z)
            addKey(obj, rotTrackX, current_frame, current_frame_local_rot.x)
            addKey(obj, rotTrackY, current_frame, current_frame_local_rot.y)
            addKey(obj, rotTrackZ, current_frame, current_frame_local_rot.z)
      
        doc.EndUndo()
        c4d.EventAdd()
      
      
    if __name__ == '__main__':
        main()
    

    Moreover looking a bit the source you probably use this http://github.com/stereolabs/zed-examples/blob/master/positional tracking/src/main.cpp for create the csv file. Which say

    initParameters.coordinate_system = COORDINATE_SYSTEM_RIGHT_HANDED_Y_UP;
    

    But I didn't succes to convert it to good thing inside c4d.
    Importing your obj with Flip X Axis enable or Swap X and Z axes give the same result and looking to your video it's seem to be the good orientation, but my matrix mathematics are sadly not good so maybe someone can help.

    About timestamp as Riccardo pointed this out, I simply make one line = one frame.


Log in to reply