We opted for Python as our programming language because it is one of the most common languages, thus it would be easier to get help with and all three team members were already comfortable and familiar with it prior to the project. Also, it has a lot of extensive third-party libraries and using APIs in our project was another key element, which python would help us with.

For script Generation, we have decide to tackle this in two different methods, Local and server LLM. For server LLM, we have opted to go with OpenAI due to past experience as a team have with the API. The user will need to put their OpenAI api into the website if they want to use the feature Moving on to local LLM, we have used LangChain and GPT4All to load our model. We have download and test multiple model from huggingface namely gpt4all-falcon and mistral-openorca which works decently with what we are trying to do

Retrieval Augmented Generation (RAG)

For RAG system, we decide to use model from huggingface community namely, "sentence-transformers/all-MiniLM-l6-v2" to transform our information which is usually in pdf or txt into vector based data. Then we can retrieve this information using LangChain and similarity search.

We evaluated several technologies when it came to audio generation, each with its own strengths and weaknesses. Our goal was to select the ones tailored specifically to our project's objectives within audio generation, them being voice cloning, easy integration with python and accurate text to speech conversion.

Text To Speech

Initially, we needed to determine our Text-to-Speech tool. Some platforms provided pre-existing voices with various specifications (e.g., Male, American or Female, Australian), while others allowed voice cloning [1]. After considering the options, we opted for IBM Watson for its pre-made voices due to its high accuracy, quality voices, reliable algorithms (e.g., processing human speech), and future-ready features such as integration with Watson Assistant and multilingualism[2]. Regarding voice-cloning, ElevenLabs emerged as our choice for its instant voice cloning capabilities and cost-effectiveness compared to alternatives[3]. Although PlayHT was considered, given its usage by PodcastAI, a thorough comparison with ElevenLabs revealed its diminished functionality and quality, coupled with a higher cost.

Audio Processing

We then needed to figure out the audio processing part, including merging files together, adding music, cutting, etc. A crucial requirement for us was the tool's capability to import and export mp3 files, rather than solely playing the sound within Python. Considering that many Python libraries offer similar audio processing functionalities, we decided to choose pydub as it seemed to cover our needs, while also being easy to use[4]. We also used moviepy to concatenate audio clips together.

Sentiment Analysis

Also, we needed a tool or a library to analyse the mood of our script to then select the appropriate music. Similarly to audio processing, the sentiment analysis libraries offered similar features[5], at least for the functions we required (accurately analysing the text sentiment). Therefore, we selected textblob as it was a library we were already familiar with and hence knew how to use effectively.

Music

In the final phase, we required a selection of at least 3 different themed soundtracks with positive, negative or a neutral mood. The decision was made to use YouTube Audio library as the platform is copyright free, fairly rich, and it is easy to find the soundtracks needed by using the 'mood' filter tab.

Even though video generation was not a priority for us as we wanted to focus on the script and the audio of our podcust, we reviewed 2 different approaches when it came to video generating - generation of a dynamic video and generation of a static video.

First Potential Solution

Initially, we wanted to utilise a text-to-video tool such as synthesia[6] to generate video clips for our episodes. The idea would be: generate video based on what the host or the guest is saying and combine it with the audio from text-to-speech model. However, after researching deeper, we realised that the text-to-video tools, including synthesia and others are really expensive while providing us with something that we do not necessarily need as people usually listen to podcasts while doing something else (chores, drive, work out)[7] and hence will not focus on the video element of the podcast. Therefore, we made a decision to do something simpler and more cost-effective.

Second Potential Solution

As a result, we implemented the static video method. The approach included two different ways of choosing the image for the static video within itself.

The first method was to generate a number of neutral themed pictures, and once the audio part is ready, select a random neutral-themed picture from the folder with these images. Then, simply play this image throughout the episode, using it as a cover for the episode. The technology we wanted to use was MidJourney (text-to-image tool) as it generated very high quality pictures, following the prompt closely[8]. The only negative it had was the absence of the api for it, therefore we would have to do more manual work and the podcast would not be as automated.

The other method of selecting an image included using a different text-to-image tool, which had an api. After the final consideration of the three best tools tailored for our needs (DeepAI, Replicate and StabilityAI), we decided to choose StabilityAI for its ease of use, well-written documentation and decent quality of images, coupled with free accessibility[9]. The video generation process included analysing the text script to identify 5-10 keywords, which would then be used to generate an image displayed throughout the episode.