*/}\n
\n Aided by text-to-image and text-to-video diffusion models, existing 4D content creation pipelines utilize score distillation sampling to optimize the entire dynamic 3D scene. However, as these pipelines generate 4D content from text or image inputs directly, \n they are constrained by limited motion capabilities and depend on unreliable prompt engineering for desired results.\n To address these problems, this work introduces 4DGen, a novel framework for grounded 4D content generation. We identify monocular video sequences as a key component in constructing the 4D content.\nOur pipeline facilitates controllable 4D generation, enabling users to specify the motion via monocular video or adopt image-to-video generations, thus offering superior control over content creation.\nFurthermore, we construct our 4D representation using dynamic 3D Gaussians, which permits efficient, high-resolution supervision through rendering during training, thereby facilitating high-quality 4D generation.\nAdditionally, we employ spatial-temporal pseudo labels on anchor frames, along with seamless consistency priors implemented through 3D-aware score distillation sampling and smoothness regularizations.\nCompared to existing video-to-4D baselines, our approach yields superior results in faithfully reconstructing input signals and realistically inferring renderings from novel viewpoints and timesteps.\nMore importantly, compared to previous image-to-4D and text-to-4D works, 4DGen supports grounded generation, offering users enhanced control and improved motion generation capabilities, a feature difficult to achieve with previous methods.\n
\n\n TL;DR: We introduce grounded 4D content generation. We identify monocular video sequences as a key component in constructing the 4D content. Our pipeline facilitates conditional 4D generation, enabling users to specify the motion (monocular videos), thus offering superior control over content creation.\n
\n {/*\n Previous work generates 4D content in one click. Our work introduces Grounded 4D Content Generation, which employs a video sequence and an optional 3D asset to specify the appearance and motion.\n
\n\n We conduct 4D generation grounded by a monocular video sequence. Our 4D scene is implemented by deforming a static set of 3D Gaussians. Besides reconstructing the input signals, we supervise our framework with spatial-temporal pseudo labels on anchor frames by a multi-view diffusion model. Moreover, we adopt seamless consistency priors implemented with score distillation sampling and unsupervised smoothness regularizations. \n
\n\n Our method is evaluated against per-frame baselines and concurrent method Consistent4D. We also provide image-to-4D results by using Stable Video Diffusion to generate a video from an image.\n
\n
\n If you want to cite our work, please kindly use:\n
\n \n {`@article{yin20234dgen,\n title={4dgen: Grounded 4d content generation with spatial-temporal consistency},\n author={Yin, Yuyang and Xu, Dejia and Wang, Zhangyang and Zhao, Yao and Wei, Yunchao},\n journal={arXiv preprint arXiv:2312.17225},\n year={2023}\n}\\}`}\n \n \n \n