Integrate the UNOMi 3D Lip Sync plugin into Blender. The approach will revolve around building an addon in Python that communicates with your existing Kaldi-based backend, applies the results (phoneme/mouth movement data) to 3D characters in Blender, and presents an intuitive UI to the user.

Overview of Requirements

• Blender version: Targeting 3.x+
• Backend: Already functional (Kaldi + Python)
• Input: Audio or text
• Output: Blendshape/morph target animation or bone movement
• Output format from backend: JSON or similar
• Goal: Apply lip sync to rigged characters in Blender (face rigs or shape keys)

Plugin Architecture

Frontend: Blender Python Addon

Backend: Existing UNOMi API (local or cloud)

Communication: REST or WebSocket (Blender → Backend)

Data Flow:

Audio/Text ➝ UNOMi API ➝ JSON Animation ➝ Blender (keyframes/blendshapes)

Development Schedule & Step-by-Step Breakdown (5 Weeks)

Week 1: Project Setup + API Integration

Step 1: Create Blender Addon Structure

• Create addon folder with __init__.py, bl_info, and registration functions
• Define basic UI panel in 3D View or Text Editor using bpy.types.Panel

Step 2: Implement Audio/Text Input

• Add UI to:
• Select or drag in audio file (.wav)
• Optionally input text for TTS (if supported)
• Save input locally in temp folder

Step 3: Send Request to UNOMi Backend

• Use requests or aiohttp to send audio or text to your backend API
• Receive JSON with animation data (phoneme timings, blendshape weights)

Week 2: Data Processing & Application

Step 4: Parse Backend Response

• Process returned JSON:
• Timecodes
• Phoneme-to-shape mappings
• Weight values over time

Step 5: Connect to Blender Character Rig

• Detect selected object and check:
• Shape keys (for blendshape-based rigs)
• Bone structure (for bone-driven rigs)
• Validate character compatibility

Week 3: Apply Keyframes

Step 6: Generate Animation in Timeline

• For Shape Keys:
• Use bpy.data.objects[...].data.shape_keys.key_blocks[...].value
• Insert keyframes using .keyframe_insert(data_path="value", frame=...)
• For Bones:
• Use bpy.data.objects[...].pose.bones[...].rotation_euler or .location
• Apply transforms and keyframes
• Optional: Smooth values with linear interpolation

Week 4: UI Polishing & Usability

Step 7: Add Playback and Preview Features

• Add “Play Preview” button in UI
• Option to delete or re-generate animation

Step 8: Export or Bake Animation

• Add option to export animation to .fbx or .glb
• Option to bake keyframes to reduce complexity

Week 5: Testing, Debugging & Packaging

Step 9: Error Handling & Logs

• Handle edge cases: missing shape keys, upload errors, backend timeout
• Display user messages via self.report({'ERROR'}, message)

Step 10: Finalize & Package Addon

• Zip plugin folder for distribution
• Write installation instructions
• Create sample character or demo .blend file

Tools and Libraries Needed

• Python (Blender native)
• requests or aiohttp for HTTP communication
• Blender API (bpy)
• Optional: NumPy or Pandas for phoneme data processing (if needed)

Deliverables

• unomi_blender_lipsync/ addon folder
• README and setup guide
• Example character with mouth rig
• (Optional) Video walkthrough/demo

Bonus Features (Post-launch Ideas)

• TTS integration if text-only input desired
• Language selector once multilingual phoneme support is live
• Batch processing for multiple characters/scenes