Getting started with building Generative AI Applications
If you are an engineer earger to get started on building Generative AI applications, the current information overload and hype cycle currently associated with Generative AI could get overwhelming for many.
Here is a list of topics that helped me get started and enjoy building Generative Ai apps.
Large and Small Language Models
Majority of the people I’ve interacted with think Generative AI = LLMs, and that one needs to go deep into understanding LLMs and how they work and how to go about training & fine tuning LLMs etc. The truth is LLMs are just one of the many components (albite and important one) of a any Generative AI application. Things have evolved to such an extent that LLMs have sort of become a commodity and many of them are available as SaaS/PaaS which are exposed as APIs such that one can simply build a React based frontend app on top of an LLM. OpenAI APIs are a classic example.
To play around with a bunch of self-hosted and local LLMs, install tools like Ollama or LM Studio where you can easily download most of the models from Llama2 to Mistral to hugginface and try them out. While Ollama provides a CLI based endpoint that you can hit to talk to an LLM, LM Studio provides a nifty chat UI interface where you cah test out the different LLMs.
Frontend UIs
If you are a frontend engineer or are looking to build a frontend app that talks to your LLMs then I would highly recommend the Vercel AI SDK and the Vercel AI templates, that allow you to clone and deploy a Gen AI app within minuties.
Prompt Engineering
Yup I know prompt engineering is a beaten horse, and people dont talk a lot about it now as compared to about 6 months back, but I can’t emphasise enough the importance of prompt engineering and the impact it has on the quality of the outputs of your Gen Ai application. Remember that with Gen AI you are building probablistic applications and the probablity of how good the apps respond to a user’s input heavily relies on the context that is passed to the LLMs via the prompt template and the prompts. All of which falls under the umbrella of Prompt Engineering.
Langchain
Over 95% of all Gen AI app tutorials that you come across would use langchain in some shape or form, and I would highly recommend investing a decent amount of time understanding langchain, you can choose either the python version or the JavaScript version of Langchain. While working with Langchain make sure you undesteand the concepts around, chains, tools, agents etc.
An Agent framework
As you move into advanced stages of building apps you’ll come across the notion of agents and multi-agent converstions, and this where there are a bunch of tools you can start looking into. Here is just a small collection of agent frameworks that have worked well for me:
Semantic Kernel
Autogen
Taskweaver
CrewAI
Common Apps to Build when learning GenAI
There is no dearth of GenAI tutorials on the internet and YouTube, and most of them are really good to get started. However, if the information overload is overwhelming, then here are a few apps to try out.
RAG based chat over a document
One of the most popular applications to build is a chatbot that answers questions about a document (usually a PDF) or a collection of documents. This is called a RAG Application, where RAG stands for Retrieval Augmented Generation. Here, you learn some key concepts on how to create chunks and embeddings from an unstructured document and store them in a Vector database. Then, you learn to query the vector database to retrieve chunks that closest match the question asked and pass it through an LLM (Large Language Model) to get a natural language output. It’s an excellent way to understand the integration of retrieval systems with generative AI.
Build a CustomGPT
OpenAI recently launched the ability to create a CustomGPT that is ‘finetuned’ or ‘trained’ to answer questions on a specific set of topics. This is a good starting point for building a no-code GenAI application. With CustomGPT, you can specialize the model on a niche subject area or style, making it ideal for creating unique applications or enhancing existing services with a tailored conversational agent. The process involves selecting relevant training data and iteratively refining the model’s responses, providing a hands-on experience with model training and fine-tuning.
Generate Artistic Images
Another exciting project is to build an application that generates artistic images based on textual descriptions using Generative AI models. This involves understanding and utilizing models like DALL-E or similar technologies. Such a project would not only be fun but also educational in terms of understanding how AI can interpret and visualize creative prompts. It’s a great way to explore the intersection of AI and creativity.
Automated Content Creation
A practical and increasingly popular GenAI application is automated content creation. This could range from generating blog articles, creating marketing copy, to even drafting code. Such applications provide an insight into how AI can assist in repetitive or creative tasks, enhancing productivity and sparking new ideas.
Personalized Recommendation Systems
Finally, building a personalized recommendation system using GenAI can be both challenging and rewarding. Such systems could recommend products, content, or even actions based on user behavior and preferences. It encompasses learning about user profiling, behavior analysis, and the application of AI in making predictive decisions.
In summary, starting with these projects can provide a comprehensive and engaging way to dive into the world of Generative AI, offering both foundational learning and practical experience.
