Hello @wickedp,
Thank you for reaching out to us. We will answer this question tomorrow more thoroughly, you happend to post after our morning meeting, but I can already see some questions we might have.
But I'm not grasping how to change from normal tangent to local space.
I am a bit confused with what you mean by normal tangent space. I assume you are referring to tangent space for a normal map. A normal map is normal (deviation) information in the space of the object it is applied to.
Vector norm_tang_to_local = ??;
The question is what you expect here to happen. Do you want to sample by a texture coordinate, then you can just use BaseShader::Sample(). Or do you want to sample by a point in world or local object space and you want to find the closest normal (driven by a normal map or not)? Your code sort of implies that you want to use your normal information called normal. Then you must simply apply the normal map sample to the vertex normal. I am assuming here that the return value of Get_NormalMapValue() is already a normal, if not, see Normal Mapping for how to convert from an RGB vector to a normal. As pseudo code:
// The vertex to sample.
const maxon::Int32 vertextIndex = 0;
// Your function, not required, everything is in object space.
const Matrix m = Get_ObjectRotMat();
// Your function, I am using this as if it would return a vertex normal.
const Vector vertex_normal = Get_PolyNormal(vertextIndex);
// Convert a coordinate in object space to texture space. This is IMHO the tricky
// part, your code implies that you got this, but it is quite hard to do IMHO.
const Vector uvw = GetTextutureCoordinate(mesh, vertextIndex);
// Your function, this has its problems as explained below. I am also using this as if
// it would sample at a point and not an area.
const Vector texture_normal = Get_NormalMapValue(uvw);
// Builds a frame, i.e., a rotation matrix/transform for the given texture normal.
const Matrix normal_transform = BuildNormalFrame(texture_normal, upVector);
// The final vertex normal tranformed by the normal map.
const Vector final_normal = normal_transform * vertex_normal;
Where BuildNormalFrame is a function which builds a frame for a vector in the common way with an input vector, an up-vector and the cross product. When the normal map is purple, the normal deviation zero, texture_normal will be the null vector and you should bail on doing further computations or just forcibly set normal_transform to the identity matrix. final_normal will then be the vertex normal rotated by the orientation stored in the normal map. E.g., when the RGB value was (128, 0, 128), i.e., the texture normal is (0, -1, 0) (as opposed to the 'default' value of (0, 0, 1) for the normal/z/k/v3 axis of a transformation matrix/frame). If then the vertex normal is (1, 0, 0), the final normal is (0, 0, 1), i.e., the vertex normal rotated 90° CCW on the Y-axis.
There is also the problem of texture sampling, which is not trivial. Your posting says Get_NormalMapValue(); // I've got this. But I would consider this the tricky part, since you will have to sample the texture at the location of the vertex which has the normal normal. The function you are using to retrieve the geometry normal is also called Get_PolyNormal(), which implies an area to be sampled and not a (vertex) point. So, Get_NormalMapValue() also samples and interpolates an area in the normal map and then returns that value? This is all quite unclear to me. There is VolumeData::GetWeights() which gives you exactly the algorithm Cinema 4D is using for retrieving a texture sample. But using this requires render-time only information. The next best option is maxon::GeometryUtilsInterface which also contains methods to convert from Cartesian (a vertex position in object space) to Barycentric coordinates (a texture coordinate). The problem is that this works on triangle level, so the uv(w) texture coordinate you get out of it is not the one you get for a whole mesh. Redoing texture sampling outside of the rendering pipeline is hard.
If you just want to get what the Cinema 4D normal map functionalities produce, you could also use texture baking.
Cheers,
Ferdinand
edit: I updated the posting.
