xVASynth v3 – CPVA Synth

Utilities








Download the main app from

the Skyrim page

.  Main app now also on

Steam

!

You can now train your own voices:

xVATrainer



List of voices available for xVASynth, from both myself and the community:

Google doc link



You can submit models at the following link, if you train them with xVATrainer:

Google forms link

Quick intro

xVASynth is an AI based app for creating new voice lines using neural speech synthesis. The app loads models individually trained on character voice data from games. The app gives users control over details such as pitch and durations of individual letters to provide control over emotion and emphasis. To see it in action, watch these short intro/tutorial videos, narrated by various supported voices:


Supported games



Discord:

https://discord.gg/nv7c6E2TzV




Patreon:

https://www.patreon.com/xvasynth



Twitter:

@dan_ruta




Preface


: The tool does not re-distribute any game assets, nor does it interact with them in any way. Game assets are used only during voice training as a reference, to guide the algorithm to drive itself to a point where it can create voices that sound similar enough to the examples. Think about it as an automated digital impersonator. Regardless, avoid using the tool in an offensive/explicit manner.

Make it obvious where you can, in descriptions that the voice samples are generated

, and are not from real human voice actors. Any issues you cause with this are on you.


Introduction

xVASynth is an AI app that generates voice acting lines using specific voices from video games. It can do text-to-speech (TTS) from text input, or voice conversion (VC) from audio input (file/microphone). Starting with v3, the app gives users artistic control over pitch, duration, energy, emotion, and style values for every letter in the audio. They also allow generating audio with explicitly defined pronunciation via ARPAbet [3] notation. Every v3 model can speak any of the 28 supported languages, and can switch between multiple languages in the same text prompt.

The use of neural speech synthesis leads to natural sounding voices, something which is very difficult to do with more traditional methods involving concatenations of existing data. It also means new vocabulary can be generated, outside of what the voice actors have already read out.


Voice Conversion (v3+)

The app can also do voice conversion, rather than text-to-speech. In this mode, you can provide a reference audio dialogue file, and the app will re-generate it but with the voice of the v3 model you select. You can provide a reference audio line by recording with your microphone (by clicking the icon), or you can drag+drop an audio file onto the icon. If needed (unlikely), you can control the voice conversion strength in the settings.


ARPAbet pronunciation (v2+)

You can specify exact pronunciation for words by using ARPAbet notation between { } brackets in the input, or by managing words in your own (or other people’s) dictionaries. Included is CMUdict with 135k words with American-English pronunciations.


NOTE


: v3 introduces several new ARPAbet symbols, for a custom extended version of the ARPAbet spec which includes sounds more typically found in other languages.

Other 3rd party dictionaries you can install into the app include:




xVADict community project – Elder Scrolls edition




:



https://www.nexusmods.com/skyrimspecialedition/mods/56778




xVADict is a community project to create ARPAbet pronunciation dictionaries, for use in xVASynth. This page contains the dictionary for the unique words found across all Elder Scrolls games.




xVADict – Alphabet Pronunciation




:





https://www.nexusmods.com/skyrimspecialedition/mods/57439




Adds the English alphabet pronunciation to xVASynth.


Batch Mode

For larger projects, where you need to synthesize a large amount of lines, you can alternatively use the Batch synthesis mode. You can use either a .txt file or a .csv file to batch generate hundreds or even thousands of lines, in one go, with parallelization. Although the pitch/duration/energy editor is sometimes needed to get a line sounding just right, it’s sometimes not needed, and this is a good way to get an initial pass on lines. Using the GPU is especially highly recommended for this, as you can greatly parallelize the number of lines generated in one go (limited by VRAM). You should also check the various settings, such as multi-threading, to get the best possible speed out of this for your system.


3D Voice embeddings visualizer

The 3D voice embeddings visualizer is an interactive panel where you can explore in 3D all the voices in the app, as seen by an AI representation learning model, projected down to 3D. There are no axes, and this serves purely as a visualization, to enable voice discovery. You can colour the points by game, or gender, and you can enable disable specific games/voices. You can load a voice by clicking it and the “Load” button, if it’s installed.



Third party plugins

The app supports third-party plugins for either/both javascript front-end (UI) and python back-end (AI) parts of the app. Plugins are a great way to customise the app to your liking, or to add new functionality to it that would be too niche or too game-specific to add to the base app for everyone. Plugins can be made for either/both the front-end/back-end of the app. Some example plugins are listed here (let me know if you make anything, and I will add it here):




Voiced Player – xVASynth Fuz Ro Bork plugin




:





https://www.nexusmods.com/skyrimspecialedition/mods/62944





A plugin to connect xVASynth up to Fuz Ro Bork, enabling xVASynth voices to be used in the Fuz Ro Bork mod.






.lip and .fuz plugin for xVASynth v2




:





https://www.nexusmods.com/skyrimspecialedition/mods/55605




A plugin to create .lip and (optionally) .fuz files automatically from audio lines generated with xVASynth, in either normal mode or batch mode, with or without multi-threading. DOES NOT NEED THE CK. Works for Skyrim, Fallout 4, Fallout 3, and Fallout New Vegas.




xVASynth plugin – Romanian Language




:





https://www.nexusmods.com/skyrimspecialedition/mods/50878




A demo plugin for v1.4.0+ of xVASynth, where third party plugins are now supported. This plugin changes the app front-end, swapping the UI language to Romanian. Full developer reference: https://github.com/DanRuta/xVA-Synth/wiki/Plugins

If you are a developer and are interested in developing a plugin, check out the documentation here:

https://github.com/DanRuta/xVA-Synth/wiki/Plugins



Nexus API integration

xVASynth has Nexusmods API integration to display what voices are available for updates/download, from any of the nexus pages listed in the “Manage Repos” sub-menu. If you have Nexus Premium, you can also download or batch download voices straight from within the app, and have them installed automatically.


App installation

You may need to install Microsoft Visual C++ Redistributable if you don’t already have it. To install the app, download it and extract it anywhere you’d like (it does not need to be in any game directory). Launch the app by double-clicking the xVASynth.exe file. If you have any issues, try running it as admin, but be mindful that Electron on Windows has some issues with drag+drop events when running as Admin.


NOTE

: v3 voices do not use a separate vocoder, as they are all-in-one models. You do not need/cannot use HiFi-GAN or WaveGlow models with v3 models



Important

:

Make sure you click “Allow” if windows asks you for permission to run the python server. I use a local HTTP server to enable communication between the python code (for the AI models) and the JavaScript code (for the Electron front-end). If there are any issues, check the server.log/app.log files (located next to xVASynth.exe) – there should be an error at the end which I’ll need to see for helping with issues.


Voice installation

The recommended way to install voices is through the Nexus API integration. However, if you don’t have Nexus Premium membership, or you’d prefer manual installation, you need to download the individual .zip files from the game-specific nexus pages (such as this one). You can either drag+drop these over the voice bar on the left in the app, or extract the voice files into the app directory, at this location: <.exe location>/resources/app/models/<game>     where <game> is the game ID. The voice .zip files already contain the required directory structure, so all you need to do is drag+drop the extracted “resources” folder from the .zip files into the folder where the xVASynth.exe file is (replacing files if prompted).

To confirm, when installing voices, you should see 3 or 4 files (a .json, a .pt, a .hg.pt, and a .wav file) all named as the voice you’re downloading, in <your xVASynth install directory>/resources/app/models/<game>/   (where <game> is cyberpunk, for models on this page).



Important

:

If you move the app files to a different directory, you MUST update the model paths in the settings, because these folder paths get initialized with the full path (starting from the drive letter) – basically, just make sure the app is looking in the new place where your models are, rather than the old folder. The app also allows you to set a different folder to store your voice models in, rather than nested in your app installation directory. The easier thing to do long-term would be to find somewhere not in your app installation folder to store your models, and set the app file paths to point there.


The voices

For Cyberpunk, the voices trained so far are as follows (“Track” the mod for updates):

v3 models:

  • V (Male)
  • Johnny
  • Judy
  • Panam
  • Kerry
  • River

Older:

  • ?

    ?


    Claire
  • ?

    ?


    Alt
  • ?

    ?


    Delamain
  • ?

    ?


    Takemura
  • ?

    ?


    Misty

  • ?


    ?


    Placide

  • ?


    ?


    Jackie

  • ?


    ?


    Elizabeth

  • ?


    ?


    Sebastian

  • ?


    ?


    Rhino









  • ?

    Rogue







  • ?

    Evelyn







  • ?

    Haru







  • ?

    Dakota







  • ?

    Gillean







  • ?

    Wakako







  • ?

    Hanako Arasaka







  • ?

    Stanley







  • ?

    Lizzy Wizzy




  • ?


    Meredith Stout





  • ?

    Maiko




  • ?


    Rachel

Where green text colour represents good quality, yellow means ok quality, and red currently quite bad (will need a good deal of playing with the input to get something good). There are several types of models and variants of models supported by the app, so I will use emojis to try to clearly label what type of model each voice is:


?


– This means the data for the voice is pre-trained using Tacotron2 [6], and the sentence structure/composition quality will be high




?



– This means the voice comes with a bespoke HiFi [4] vocoder model, meaning the audio quality will be high










– This means the voice model is FastPitch1.1, enabling energy control, speech-to-speech, and ARPAbet pronunciation. Tacotron2 isn’t needed for this. (rad icon for RAD-TTS the built-in alignment mechanism replacing Tacotron2)



Note: To start with, most voice models will be v1.0 FastPitch, but they will eventually all be re-trained with the better v2.0 models with all the new features. I have over 425 voices to get through, so it may take a while.

You can optionally install WaveGlow [5] models from

here

, for extra vocoder options, but these are much slower, and almost always not as good as HiFi-GAN.


Tips

The most important thing to keep in mind is to make sure to play around with the editor, to get the best quality from the generated lines. If some words/letters sound bad, try changing the pitch/duration/energy values. Tinny artefacts can normally be fixed by slightly shortening the durations of offending letters. If you absolutely can’t get it to say it well, and ARPAbet pronunciation doesn’t help, try re-wording the line.

Check out the community guide here, where anyone can add their tips/advice for how to get the best quality out of the tool:

https://github.com/DanRuta/xvasynth-community-guide

You can also access this from the info (i) menu in the app.


Downstream uses


If you make anything with this tool (mod or otherwise), let me know and I will include it here.

YouTube playlist of xVA experiments (WaveGlow MaleSlyCynical):

https://www.youtube.com/playlist?list=PLDGgH-fuVvfa8-HFdSi7ls1ykLIuquIpD







Radio New Vegas GPT-3:






https://bunglepaws.neocities.org/radio_new_vegas_gpt3.html











[Fallout 4] Fallout 4 Point Lookout – Voiced Player Lines Addon




:



https://www.nexusmods.com/fallout4/mods/60387




XVASynth generated voice lines for Nate and Nora in the Fallout 4 Point Lookout mod.



[Fallout 4] Flashy(JoeR) – Gun For Hire – Commonwealth Mercenary Jobs



:





https://www.nexusmods.com/fallout4/mods/49610




Gun For Hire allows you to open a business outside of Diamond City and to run never-ending jobs for clients from a base of 27 different archetypes.






[Skyrim] Auto Sleep For Me Now




:



https://www.nexusmods.com/skyrimspecialedition/mods/56850




The most vanilla follower detect player auto sleep ever






[Skyrim] Sit For Me Now




:



https://www.nexusmods.com/skyrimspecialedition/mods/57423




Your follower auto sits with you




[Skyrim] Me So Hungry




:





https://www.nexusmods.com/skyrimspecialedition/mods/57184




NPCs cooking




[Fallout New Vegas] Easy Lanius




:





https://www.nexusmods.com/newvegas/mods/74565



Replaces Easy Pete’s voice with a Lanius voice created by xVASynth2.

The Monster of the East has come to Goodsprings to retire.




[Skyrim] Teldryn Sero Dialogue Expansion




:





https://www.nexusmods.com/skyrimspecialedition/mods/42434



More unique dialogue for Teldryn Sero — Adds conditional commentary and some player dialogue options.




[Oblivion] Oblivion Nouveau Uncut




:





https://www.nexusmods.com/oblivion/mods/47191




Adds over 50 npcs, 6 side quests & 1 questline, voiced Nord & Redguard guards, new locations & items, over 1000 lines of dialogue and much more!



[Oblivion] Chapter II – Daggerfall 3E433



:



https://www.nexusmods.com/oblivion/mods/49031




Welcome to the Kingdom of Daggerfall, Experience the entirety of Hammerfell & High Rock recreated lore friendly with Chapter II Content







[Fallout 4] Isran (Skyrim) Male Protagonist Voice Replacer



:





https://www.nexusmods.com/fallout4/mods/56972





Bored with the generic male protagonist voice? Don’t like him saying sarcastic lines in an exaggerated way? Want to make a tough-looking character but the voice just doesn’t cut it for you? This mod is for you! It replaces the male protagonist’s voice with Isran’s voice, making your character actually sound tough and threatening (and serious).



[Skyrim] The Courier Crew



:



https://www.nexusmods.com/skyrim/mods/110453




Adds two additional couriers w/ all courier voice lines for a total of three couriers




[Cyberpunk 2077] All Rhino All the Time



:







https://www.nexusmods.com/cyberpunk2077/mods/2826




Complete AUDIO AND VISUAL overhaul for various characters into the large muscular beauty, Rhino. Som (Published on: 2023-08-31 22:19:00)

Related Game Mods

Attached Files:

File Name Download Count Download Button
Alt-2162-1-0-1624955178.zip 0 Download
Claire-2162-1-0-1623689050.zip 0 Download
Dakota-2162-1-0-1647208262.zip 0 Download
Delamain-2162-1-0-1624955228.zip 0 Download
Elizabeth-2162-1-0-1628871256.zip 0 Download
Evelyn-2162-2-0-1644162937.zip 0 Download
Gillean-2162-1-0-1647208320.zip 0 Download
Hanako-2162-2-0-1648576108.zip 0 Download
Haru-2162-2-0-1644163011.zip 0 Download
Jackie-2162-1-0-1628022354.zip 0 Download
Johnny-2162-3-0-1693516366.zip 0 Download
Judy-2162-3-0-1693516536.zip 0 Download
Kerry-2162-3-0-1693516622.zip 0 Download
Lizzy Wizzy-2162-2-0-1651402159.zip 0 Download
Maiko-2162-2-0-1655050614.zip 0 Download
Meredith Stout-2162-2-0-1651402213.zip 0 Download
Misty-2162-1-0-1627205967.zip 0 Download
Panam-2162-3-0-1693516710.zip 0 Download
Placide-2162-1-0-1627206020.zip 0 Download
Rachel-2162-2-0-1655050678.zip 0 Download
Rhino-2162-1-0-1631306999.zip 0 Download
River-2162-3-0-1693516792.zip 0 Download
Rogue-2162-2-0-1641136448.zip 0 Download
Sebastian-2162-1-0-1630340366.zip 0 Download
Stanley-2162-2-0-1648576184.zip 0 Download
Takemura-2162-1-0-1625762814.zip 0 Download
V (Male)-2162-3-0-1684850896.zip 0 Download
Wakako-2162-1-0-1647208389.zip 0 Download

Leave a Reply

Your email address will not be published. Required fields are marked *