Show HN: Lipdub Videos to Any Language https://ift.tt/OJ5btHx

Show HN: Lipdub Videos to Any Language TLDR: Translate, dub, and lip-sync videos into 29+ languages. Here's an example: https://ift.tt/vYwO8Xh... I was creating a few educational math videos for kids in Latin America last summer, and I noticed they weren't easily following subtitles. I wanted to make the video content more engaging for them, but I'm still a ways away from Spanish proficiency. Went down the rabbit hole of audio + video AI models... and now we have the initial version of Viva Labs! Given a video, we first translate the audio to the target language using a pipeline of ASR models, translation APIs, & LLMs - users can edit translations to fix any errors. Then, we dub the audio with similar voices or voice clones. Finally, we sync the speakers lip movements to match the dubbed audio. Our early users have surprised us with some of the common use cases like dubbing online course content, product explainers + marketing promos, podcasts, and international newscasts Still a lot of work to do to create full immersion. The following areas of technical exploration we're pursuing: 1) matching the output audio's tone, prosody, & emotion with that of the input voice using speech to speech models, 2) training lip sync models with more robust lip sync for extreme poses and occluding objects, 3) using LLMs to produce translations that are more colloquial and more closely match the pacing of original audio. Here's a sample of Mira Murati's GPT4o announcement lip dubbed to Russian: https://ift.tt/vYwO8Xh... Looking forward to seeing what videos you dub! You can dub 3 minutes of video for free at app.vivalabs.ai or you can ask our twitter bot x.com/VivaDubs to audio dub a video on twitter https://app.vivalabs.ai June 18, 2024 at 03:01AM

Komentar

Posting Komentar

Postingan populer dari blog ini

Show HN: Interactive exercises for GNU grep, sed and awk https://ift.tt/OxeFwah

Show HN: Interactive exercises for GNU grep, sed and awk Hello! For the past few months, I've been using a Python framework called Textual to create TUI apps for interactive exercises. Released the app for GNU awk earlier today, so thought I'd create a post here. If you already know how to manage Python packages, you can use the following command to get all the three apps: pip install grepexercises sedexercises awkexercises `pipx` should also work, but I haven't tested it. The GitHub repo has the source code as well as more detailed installation instructions. You can use alternative CLI tools to solve these exercises as well. For example, Perl instead of GNU awk or ripgrep instead of GNU grep and so on. Hope you find these TUI apps useful. I'd highly appreciate your feedback. Happy learning :) https://ift.tt/Hp9kFEi August 17, 2023 at 05:13PM
Baca selengkapnya

Show HN: Create demos & guides just with a simple prompt https://ift.tt/HfWo3mz

Show HN: Create demos & guides just with a simple prompt Teams often create demos & guides on a daily basis. But creating them is time consuming and needs a lot of effort. From content to the branding, customizations, and many more - they are not something you would want to do on a daily basis. Enter Fable. Purpose built for teams in marketing, sales & customer success. Fable is apt for engaging visitors, closing deals faster, and onboarding your users quickly. https://ift.tt/IVCbkiG October 17, 2024 at 08:32PM
Baca selengkapnya