0

I have some footage, which I've tracked in After Effects. I've exported that 3D tracked camera to C4D Lite. I then took photos of the filming location and used Meshroom (a photogrammetry tool) to generate a mesh from the photos.

Now I want to line up the photogrammetry mesh in C4D with nulls that I've placed relative to the tracking points in After Effects. What's the best way to do this? Or is there a better workflow to get a photogrammetry mesh lined up with tracked footage?

More info: Specifically, after watching Corridor's real-life portals video I decided to try to create my own portal effect. My plan is to take a rough photogrammetry mesh of the filming area. I would then track the camera in After Effects, somehow get that tracking data into a 3D modelling program, load the photogrammetry mesh in, line everything up so the tracked camera moves through the mesh as it does in the footage then rotate the camera and its animation 180° around the point where the portal mask is tracked onto the footage in AE. This would create a perfect replica of the camera's original motion, but looking out at the world from a different angle.

As such I can't just use the footage as the photogrammetry input, as I specifically need the photogrammetry mesh to include unseen areas of the world.

  • Ideally you would have used images of the footage in mushroom too then your solutions would converge and the locations of the camera positions would have been solved in one go. (photogramettery is sort of automated tracking) – joojaa Jan 11 at 8:19
  • @joojaa I'm not really sure what you mean. I'd still just get a mesh with no particular way to line it up with the camera tracking from AE. As in, I would have no way to get the scale, rotation and position of the mesh correct relative to the tracked camera. – Clonkex Jan 11 at 9:01
  • No you get a mesh and postion of the cameras. See mushroom solves the camera position in the first pass. – joojaa Jan 11 at 9:08
  • @joojaa Well sure it solves the camera positions, but how would that be useful? I'm just not seeing the practical connection. I can't export the camera positions, and even if I could, I couldn't easily turn those positions into an animated camera in a 3D modelling program or After Effects. Plus the results of photogrammetry from video would be far inferior to those from photos, since sharp images give drastically better results. – Clonkex Jan 11 at 10:10
  • Positions are animation – joojaa Jan 11 at 10:11
0

In the end I decided to just learn how Blender's scripting works and wrote a script. I know there would be simpler ways to calculate the transformations but I'm no maths expert and I wanted to spend my time debugging my logic errors rather my maths errors.

Basically what this script does is take 3 input points (in the form of empties) relative to the photogrammetry mesh and 3 input points in the same position relative to the tracking data, then attempts to line them up with each other, bringing the photogrammetry mesh along with it. This actually works pretty well in my testing. It's obviously not the fastest workflow but it does work and that's the important part.

import bpy
import bmesh
from mathutils import Matrix

#####################################
# TO USE:
#
# 1) create empties at three easily-identified positions in tracking data, called ea1, ea2 and ea3
# 2) create empties at the same three positions but in photogrammetry data, called eb1, eb2 and eb3
#    NOTE: eb1, eb2 and eb3 must be positioned IN THE SAME ORDER as ea1, ea2 and ea3!
# 3) name the photogrammetry mesh "photogrammetry"
# 4) run the script and be amazed
#
#####################################

# from https://blender.stackexchange.com/a/118783/41192
def createLookRotation(forward, up):
    rot = Matrix.Identity(3)
    rot[0] = up                 # x
    rot[1] = forward.cross(up)  # y
    rot[2] = forward            # z
    rot = rot.transposed()
    return rot.to_quaternion()

def doMeshThing(obj, mesh, empty1, empty2, empty3):
    bm = bmesh.new()
    bm.verts.new(empty1.location)
    bm.verts.new(empty2.location)
    bm.verts.new(empty3.location)
    bm.verts.ensure_lookup_table()
    bm.faces.new([bm.verts[0], bm.verts[1], bm.verts[2]])
    bm.normal_update()
    bm.to_mesh(mesh)
    bpy.ops.object.select_all(action='DESELECT')
    obj.select = True
    bpy.ops.object.origin_set(type='ORIGIN_GEOMETRY')
    bm.faces.ensure_lookup_table()
    normal = bm.faces[0].normal
    guide = empty2.location - empty1.location
    quatRot = createLookRotation(normal, guide)
    obj.rotation_mode = 'QUATERNION'
    obj.rotation_quaternion = quatRot.inverted()
    bpy.ops.object.transform_apply(location=False, rotation=True, scale=False)
    obj.rotation_quaternion = quatRot
    bm.free()

ea1 = bpy.data.objects['ea1']
ea2 = bpy.data.objects['ea2']
ea3 = bpy.data.objects['ea3']
eb1 = bpy.data.objects['eb1']
eb2 = bpy.data.objects['eb2']
eb3 = bpy.data.objects['eb3']

mesha = bpy.data.meshes.new('mesha')
meshb = bpy.data.meshes.new('meshb')

obja = bpy.data.objects.new('obja', mesha)
objb = bpy.data.objects.new('objb', meshb)

scene = bpy.context.scene
scene.objects.link(obja)
scene.objects.link(objb)

doMeshThing(obja, mesha, ea1, ea2, ea3)
doMeshThing(objb, meshb, eb1, eb2, eb3)

bpy.context.scene.update()

# position and rotate objb (photogrammetry) to match position and rotation of obja (tracking)
photogrammetryMesh = bpy.data.objects['photogrammetry']
photogrammetryMesh.parent = objb
photogrammetryMesh.matrix_parent_inverse = objb.matrix_world.inverted()

objb.location = obja.location
objb.rotation_quaternion = obja.rotation_quaternion

# calculate average dimension difference here and scale
scaleDiffX = obja.dimensions.x / objb.dimensions.x
scaleDiffY = obja.dimensions.y / objb.dimensions.y
scaleDiffZ = obja.dimensions.z / objb.dimensions.z
#averageScaleDiff = (scaleDiffX + scaleDiffY + scaleDiffZ) / 3
averageScaleDiff = (scaleDiffX + scaleDiffY) / 2 # seems to be in local space, so z is useless
objb.scale.x = averageScaleDiff
objb.scale.y = averageScaleDiff
objb.scale.z = averageScaleDiff

bpy.context.scene.update()

parented_wm = photogrammetryMesh.matrix_world.copy()
photogrammetryMesh.parent = None
photogrammetryMesh.matrix_world = parented_wm

bpy.ops.object.select_all(action='DESELECT')
obja.select = True
bpy.ops.object.delete()
bpy.ops.object.select_all(action='DESELECT')
objb.select = True
bpy.ops.object.delete()
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.