we will answer this more thoroughly tomorrow, but the trick when providing a vertex color map in a cache, is to select it, as this is the condition for Cinema to draw it. For your "moved goal posts" of having a shader/material, this can become tricky, because as providing materials is not an intended functionality for generator caches. But you probably know that.
The common workaround is then to assign the material to the generator and not the cache and hide the material as you already said yourself. But when you are going to rely on vertex colors, you are going to have a problem, I think. The problem is that you cannot assign the vertex color tag to the generator, as this won't work. Vertex color tags seem to be unable to reach into caches, I just tried. So, you will have to keep the vertex color tag on the cache. You can then still create just your material with a vertex map shader in it to do all your fancy shading and assign it to the generator (and hide it). The problem is: The vertex color shader would then have to reach/link into the cache to get the tag from there. Which I think should work in principal, have not tried myself though, but is obviously the definition of "dicey".
PS: This is a "private" answer of mine, just because I thought it is an interesting problem ^^
# Pretty straight forward per vertex definition of vertex colors.
count = node.GetPointCount()
tag = c4d.VertexColorTag(count)
data = tag.GetDataAddressW()
for pid in range(count):
color = c4d.Vector4d(random.uniform(0., 1.),
random.uniform(0., 1.), 1.)
c4d.VertexColorTag.SetPoint(data, None, None, pid, color)
# Add the tag and the magic bit, quite literately, select it.
# Get our cache source, cause I was to lazy to come up with some geom. :)
source = op[c4d.ID_USERDATA, 1]
if not isinstance(source, c4d.PolygonObject):
# Clone the cache source and create a vertex color map for it.
cache = source.GetClone(0)