mirror of
https://github.com/huggingface/diffusers.git
synced 2026-06-02 00:01:34 +08:00
fc77592427
* feat: add Motif Video T2V and I2V pipelines with AdaptiveProjectedGuidance support Add complete Motif Video implementation to diffusers: New Models: - Add MotifVideoTransformer3DModel with T5Gemma2Encoder for multimodal conditioning - Supports text-to-video and image-to-video generation with vision tower integration New Pipelines: - Add MotifVideoPipeline for text-to-video generation - Default resolution: 736x1280, 121 frames, 25 fps - Supports classifier-free guidance and AdaptiveProjectedGuidance - Add MotifVideoImage2VideoPipeline for image-to-video generation - First frame conditioning with vision encoder - Same defaults as T2V pipeline Enhanced Guidance: - Update AdaptiveProjectedGuidance with normalization_dims parameter - Support "spatial" normalization for 5D tensors (per-frame spatial normalization) - Support custom dimension lists for flexible normalization - Update AdaptiveProjectedMixGuidance with same parameter Documentation & Tests: - Add comprehensive API documentation for transformer and pipelines - Add test suites for both T2V and I2V pipelines - Register all new components in __init__ files - Add dummy objects for torch and transformers backends Total: 18 files changed, 3416 insertions(+), 2 deletions(-) * Remove linear quadratic * Remove musicldm * Update docstring * Address vision_encoder comment * Add copy source in I2V pippeline * Refactor _get_prompt_embeds Co-authored-by: Beomgyu Kim <beomgyu.kim@motiftech.io> * Fix a typo * Refactor MotifVideo transformer to use diffusers Attention conventions - Use default Attention class with custom MotifVideoAttnProcessor2_0 - Inline cross-attention in transformer blocks - Use dispatch_attention_fn for backend support - Inherit AttentionMixin for attn_processors/set_attn_processor - Move TransformerBlockRegistry to _helpers.py - Add _repeated_blocks for regional compilation * Use base classes for scheduler and guider * Implement MotifVideoAttention * Update style and quality * Fix a typo * Fix a typo * Fix a typo * Update year * Address rope dtype * Update docstring and remove frame_rate * Address unused sigmas * Add available processors * Address copy from comment * Remove torch.no_grad() * Remove use_attention_mask * Address inline cross-attention * Address compute dtype * Remove unused variables * Merge main APG into this branch and update documentation * Refactor cross attention processor * Remove unused timestep * Inline create_attention_mask * Make guider required * Address encode_prompt comment * Address preprocess_video comment * Use T5Gemma2Encoder in test cases * Address None feature_extractor * Address output type * Renable skipped tests * Update style and quality * Generate standard transformer test case * Add model test case * Remove guider in documentation * Implement cross_attn layer * Remove prepare_negative_prompt * Address latent is None * Clean up feature_extractor * Fix prepare_latents * Remove transformers assertion * Fix style and quality * Fix python utils/check_copies.py --fix_and_overwrite python utils/check_dummies.py --fix_and_overwrite outputs * Add dropout rate to text config * Skip tests requiring guidance_scale * Fix encode_prompt in test cases * Fix test_cpu_offload_forward_pass_twice * Update tests/pipelines/motif_video/test_motif_video.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Update tests/pipelines/motif_video/test_motif_video.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Update tests/pipelines/motif_video/test_motif_video.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Update tests/pipelines/motif_video/test_motif_video_image2video.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Address test_attention_slicing_forward_pass comment * Update tests/pipelines/motif_video/test_motif_video_image2video.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Update tests/pipelines/motif_video/test_motif_video_image2video.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Update tests/pipelines/motif_video/test_motif_video_image2video.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Skip I2V test cases * Fix style and quality * Add docs to toctree * Fix docs location in toctree and add link in overview * Inline gradient checkpointing * Add _keep_in_fp32_modules for timestep_embedder * Address num_decoder_layers comment * Address guider is not None comment * Remove _keep_in_fp32_modules * Address parameter_dtype comment --------- Co-authored-by: Ken Cheung <ken.cheung@motiftech.io> Co-authored-by: Beomgyu Kim <beomgyu.kim@motiftech.io> Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> Co-authored-by: YiYi Xu <yixu310@gmail.com>