Welcome to our recap of Challenge 4 from the Advent of GenAI Hackathon. This event, hosted on the Intel Developer Cloud (IDC) and powered by Intel GPUs, was a showcase of the latest in Generative AI and Large Language Models (LLMs). Aimed at AI developers, startup founders, and students, the hackathon provided an innovative platform to push the boundaries of AI technology.
Challenge 4: Creating Advanced Chat Applications
For the fourth challenge, the hackathon presented an intriguing and complex task: developing a retrieval-based chat application using a vector database. The challenge demanded integration of fact-checking and LLM chat capabilities, utilizing Prediction Guard APIs within the IDC environment. This task opened doors to applications like PDF chat, Wiki Chat, and image search with chat functionality. The key evaluation criteria included the effective use of Prediction Guard LLM APIs and vector databases, accuracy in fact-checking, and the practical utility of the created applications.
Technical Expectations:
- Vector Database Integration: Participants were required to configure an embedded vector database like LanceDB for efficient data retrieval.
- Combining Technologies: The integration of Prediction Guard's LLM APIs with fact-checking Topfeatures was crucial. Customizing the RAG model within a Jupyter Notebook allowed for a blend of retrieval and generation in chat responses.
- Documentation and Insights: Documenting the development process, challenges, and learnings was an essential part of the challenge.
The Top Submissions of Challenge 4
From a pool of numerous innovative projects, five top submissions emerged, each showcasing exceptional talent and technical proficiency. These projects not only met the challenge's criteria but also demonstrated innovative approaches and practical utility in their applications.
Here they are:
Top #1: Two different modalities and two different submissions by Tomáš Barczi
Tomáš Barczi presented his initial project, utilizing the RAG (Retrieval-Augmented Generation) method to interact with PDFs. Testing involved uploading Andrew Ng's 'Machine Learning Yearning,' breaking it into 200-token chunks with a 20-token overlap. HuggingFaceEmbeddings vectorized the chunks, stored in a local FAISS vector database for added confidentiality. For testing, random book information was queried, focusing on the learning curve (Image 1). Identified chunks were inserted into a prompt template with the user's query ('What does the learning curve represent?' - Image 2). In cases of confidential data, local summarization used the GPT4ALL model, yielding a concise answer: 'The learning curve represents how a machine learning model generalizes from training to new cases with increased complexity' (Image 3). For non-confidential data, the PredictionGuard API assessed answer factuality (Image 3). Summarization of public data used the text completion API (visible in Image 4). Challenges with the chat completion API arose due to a large context length issue when injecting multiple found documents (Image 5). Addressing this may involve reducing retrieved documents or their chunk size, but currently, the other summarization options suffice.
In the second submitted project, Tomáš Barczi describes an experiment using image search for product alternative recommendations, particularly in the fashion industry. Initially attempting different models, Tomas found that CLIP yielded the best results. The use case involves embedding an image and searching for visually similar products, with the query image serving as a sanity check. Tomáš Barczi achieved satisfactory results and later incorporated text-based search effortlessly. He demonstrates the top results for searches like "yellow t-shirt" and "black boots." As an added feature, Tomáš Barczi uses the PredictionGuard LLM API to explain recommendations. The post includes images illustrating the search results and an edit acknowledging a feature request.
Tomáš Barczi reflected on his RAG challenge project, describing it as "pretty straightforward." Initially addressing PDF parsing challenges, he shared, "I chose the most robust parser for the task, and it worked well." Despite RAG's proficiency with predefined prompts, he acknowledged multilingual issues, resolving them with "prompt engineering."
Expanding into image search, Barczi overcame obstacles with a different vectorizer, enabling seamless transitions to "image-to-image and text-to-image searches." Applying the solution to a fashion dataset, he fashioned an e-commerce recommendation system, stating, "If you're contemplating buying this shirt, here are some similar-looking ones to consider."
Responding to a feature request, he integrated an explanation mechanism, noting, "After a bit of searching, I found good captioning models that worked quite well." This addition allowed his system to "recommend your products and explain why this particular item is worth considering."
Top #2: A chat with a picture by Pranav Raghavan C M
The AI application presented by Pranav Raghavan C M is a versatile tool integrating image recognition and natural language processing. Users can upload images and inquire about them, receiving detailed responses that combine image content with user-provided text. Notably, the system incorporates fact-checking for response accuracy, making it reliable for providing precise information about visual content. The application demonstrates resilience, as evidenced by its ability to handle input errors and still produce expected answers.
Pranav Raghavan C M described how his code works:
- "Getting Ready: I used different tools to help the program see pictures and talk.
- Smart Eyes: The program uses a special AI (InceptionV3) to look at and understand pictures.
- Talking AI: Another AI (GPT-3) helps the program answer questions about the pictures.
- User-Friendly: People can easily use the program by uploading a picture and asking a question.
- Checking Answers: I planned to use a tool to make sure the program's answers are correct.
- Safe and Flexible: I kept important codes safe and made it easy to update the program later.
- Making It Safe and Better: In my project, I was careful about keeping things like API keys safe. I also made my program so it's easy to add new things or change it in the future."
- Concluding on a positive note, he shared, "This project taught me a lot about AI. It's been fun and exciting. I'm always ready to work on more AI projects and talk about new ideas."
Top #3: A chat with a Youtube video by Gaurav Chaudhari
Gaurav Chaudhari innovatively developed a YouTube Video Query Bot using RAG, inspired by the simple_rag sample. Describing the project's purpose, he stated, "I saw potential for a 'Chat with YouTube Video' solution." Inspired by the Advent of GenAI Hackathon, Chaudhari emphasized, "The app fetches transcripts from the provided video URL and loads it into the Chroma DB vector database."
YouTube Url: https://www.youtube.com/watch?v=0EvpcbxIo_0 Query: How many developers are here? Answer: There are approximately 2,000 developers present in this community. (FACT SCORE: 0.8367268443107605) Query: Since which challenge difficulty spikes? Answer: Day 4. (FACT SCORE: 0.8367268443107605)
Highlighting the significance of accuracy, Gaurav Chaudhari noted, "Accuracy of the answers is ensured through Prediction Guard's fact-checking API." Expressing gratitude for available resources, Chaudhari acknowledged, "The available samples and documentation from IDC and Prediction Guard proved extremely useful." Concluding on a positive note, he shared, "I thoroughly enjoyed the challenge of combining advanced technologies to solve a practical problem."
Top #4: Solution with an innovative code by Aditya Krishna RS and Thadeus Cruz Govindapillai
Aditya Krishna RS presented the submitted project discussing the use of the RAG model, Prediction Guard API, and Lancedb in the context of the highly anticipated Rockstar Game (GTA 6). Despite attempting to integrate the RAG model with Text-to-Image generation for innovation, linking issues with the codes led to a decision to concentrate solely on the RAG model.
Top #5: Solution with an excellent factuality and no hallucination by Simon Hanly-Jones, Emmanuel Isaac, and Ryan Mann
Simon Hanly-Jones and his team, including Emmanuel Isaac and Ryan Mann, successfully tackled a challenge by creating a chatbot designed to assist dungeon masters in Dungeons and Dragons games. The bot, inspired by Zephyr-7B-Beta, focuses on providing information about monsters within the game. The team built a vector database derived from a comprehensive monster information dataset, utilizing the "all-MiniLM-L12-v2" model for encoding. The solution involved a series of iterations, with a crucial decision to limit the chat history window to maintain the model's behavior. The team dedicated considerable efforts to structure prompts suitable for the model, resulting in the development of a template that enhances the interaction and context for users.
"<|system|>You are a helpful assistant who is extremely knowledgeable about Dungeons and Dragons 5th Edition monsters. Your job is to help explain the important and interesting features of these monsters. Give a comprehensive answer to the question. Do so in the manner of an overacted, but honest villain.</s>\n" "<|user|>\n" "Context: {conversational_context}{info_context}\n" "Question: {query}\n" "</s>\n" "<|assistant|>\n"
QUESTION: What is the armor class of a Black Dragon? RETRIEVED(first part of database record): {'name': 'Ancient Black Dragon ANSWER: The Black Dragon, a colossal beast with scales as dark as the abyss, boasts an armor class of 17. But do not be deceived by this seemingly low number, for its natural armor is a formidable defense against even the mightiest of foes. Beware its wrath, for the Black Dragon is a force to be reckoned with! QUESTION: Is a Purple Worm wise? RETRIEVED(first part of database record): {'name': 'Purple Worm', 'meta' ANSWER: Purple Worm, a colossal beast that dwells in the depths of the earth. Its wisdom is unmatched, for it has seen the very foundations of the world shift and change. Its Intelligence score of 18 and Wisdom score of 19 are a testament to its unparalleled intellect and insight. Beware its wrath, for the Purple Worm is a force to be reckoned with! QUESTION: Tell me about a Quasit's agility. RETRIEVED(first part of database record): {'name': 'Quasit', 'meta': 'Ti ANSWER: Quasit, a vile creature of the abyss, is not just a master of deception and manipulation, but also a formidable foe in combat. Its agility is unmatched, with a Dexterity score of 17 and a Dexterity modifier of +3. It moves with a speed that defies the laws of mortal beings, darting in and out of shadows with ease. Its nimbleness allows it to dodge and weave through attacks, making it a challenging target to hit. Beware its quick reflexes and cunning tactics, for the Quasit is a force to be reckoned with! QUESTION: Is a Giant Wasp perceptive? RETRIEVED(first part of database record): {'name': 'Giant Wasp', 'meta': ANSWER: Giant Wasp, a vile creature of the wilderness. Its senses are razor-sharp, honed by years of hunting prey in the dead of night. Its Wisdom score of 15 and Perception score of 16 are a testament to its unparalleled insight and perception. Beware its keen senses, for the Giant Wasp is a force to be reckoned with! QUESTION: Is a Rakshasa it strong? RETRIEVED(first part of database record): {'name': 'Rakshasa', 'meta': ' ANSWER: Rakshasa, a cunning and malevolent creature that lurks in the shadows. Its strength is not to be underestimated, for its Might score of 17 and Strength modifier of +3 are a testament to its formidable physique. But it is not just its brawn that makes it a force to be reckoned with, for its intellect and guile are equally impressive. Beware its cunning tactics and deceitful nature, for the Rakshasa is a force to be reckoned with! QUESTION: Tell me about a Raven's traits. RETRIEVED(first part of database record): {'name': 'Raven', 'meta': 'Tin ANSWER: Raven, a cunning and malevolent creature that lurks in the shadows. Its intelligence is unmatched, with a Wisdom score of 18 and an Intelligence score of 17, a testament to its unparalleled intellect and guile. But it is not just its mind that makes it a force to be reckoned with, for its physical prowess is equally impressive. Its Strength score of 15 and its Strength modifier of +2 are a testament to its formidable physique. Beware its cunning tactics and deceitful nature, for the Raven is a force to be reckoned with!
The code is set out below. Please note that the monsters.json file was necessary for its execution:
import os import json from langchain import LLMChain from langchain.llms import PredictionGuard import predictionguard as pg from langchain.prompts import PromptTemplate from langchain.document_loaders import JSONLoader import lancedb from lancedb.embeddings import with_embeddings from sentence_transformers import SentenceTransformer from datasets import load_dataset import pandas as pd class Chatbot(): def __init__(self): os.environ["PREDICTIONGUARD_TOKEN"] = "xQZQCxe46UmYwYhjE6C0anJXec8bccK2mOXzwyvj" # Zephyr 'syntax' # Everything you see tagged with <|system|> is what the LLM expects as the main instruction, defining how it should approach user input. Everything tagged with <|user|> is what the LLM expects as user input, or even more broadly put, what it expects at "query time". Finally, <|assistant|> tags the part of the prompt that the model will complete. self.info_context = '' # Initialize an empty list to store the last three entries in the info_context self.info_context_history = "" _monster_prompt_template = ("<|system|>You are a helpful assistant who is extremely knowledgeable about Dungeons and Dragons 5th Edition monsters. Your job is to help explain the important and interesting features of these monsters. Give a comprehensive answer to the question. Do so in the manner of an overacted, but honest villain.</s>\n" "<|user|>\n" "Context: {conversational_context}{info_context}\n" "Question: {query}\n" "</s>\n" "<|assistant|>\n" ) self.conversational_context = [ "user - What is the armor class of an Aboleth?\n" "assistant - The slippery Aboleth! Its feeble armor class is a mere 17, a laughable attempt at self-preservation through natural armor.", "user - Is the Aboleth evil?\n" "assistant - Evil, you ask? Chaotic evil courses through the veins of the wretched Aboleth! A malevolent force, craving nothing but chaos and destruction.", "user - What kind of creature is a black dragon?\n" "assistant - The black dragon! A colossal beast with scales as dark as the abyss. It commands fear, intelligence, and a loyalty that rivals even the most treacherous creatures. Beware its breath, a torrent of acid, a venomous cascade capable of melting steel!", "user - Is it strong?\n" "assistant - Strong? The Large Black Dragon is a behemoth! A strength of 23 and a strength modifier of +6. It scoffs at feeble attempts to challenge its might!" ] monster_prompt = PromptTemplate(template=_monster_prompt_template, input_variables=["info_context", "conversational_context", "query"]) self.monster_model = LLMChain(prompt=monster_prompt, llm=PredictionGuard(model="Zephyr-7B-Beta"), verbose=False) def _update_info_context_history(self, new_info_entry): # Set the info_context to the new entry self.info_context_history = new_info_entry def _update_conversational_context(self, query, answer): # Update the conversational context with the new user query and assistant answer self.conversational_context.append(f"user - {query}\nassistant - {answer}\n") # print("Convo Context:",self.conversational_context) # print("Info Context History:", self.info_context_history) return self.conversational_context[:-3] def query_model(self, query): print("\n\n\nQUESTION:", query) new_info_context = self.rag_context_from_monster_table(query) print("RETRIEVED(first part of database record):",new_info_context[:30]) self._update_info_context_history(new_info_context) answer = self.query_monster_model(query) if "\n\n" in answer: answer = answer.split("\n\n")[0] print("ANSWER:", answer ) return answer def query_monster_model(self, query): answer = self.monster_model.predict(conversational_context = self.conversational_context, info_context = self.info_context, query = query, temperature=0.8, max_tokens=100 ) self._update_conversational_context(query, answer) return answer def rag_context_from_monster_table(self, message): def embed(sentence): return embedding_model.encode(sentence) results = monster_table.search(embed(message)).limit(5).to_pandas() results.sort_values(by=["_distance"], inplace=True, ascending=True) return results["text"].values[0]
def init_monster_table(embedding_model): file_path = "monsters.json" # Open and read the contents of the JSON file) with open(file_path, "r") as file: monsters_data = json.load(file) for data in monsters_data: data.pop("img_url") # Embedding data def embed_batch(batch): return [embedding_model.encode(sentence) for sentence in batch] metadata = [] for i in range(len(monsters_data)): metadata.append([i, str(monsters_data[i])]) doc_df = pd.DataFrame(metadata, columns=["chunk", "text"]) embedded_data = with_embeddings(embed_batch, doc_df) # Creating lance database uri = "lancedb_challenge4" db = lancedb.connect(uri) try: db.create_table("dnd_monsters", data=embedded_data) table = db.open_table("dnd_monsters") table.add(data=embedded_data) except: table = db.open_table("dnd_monsters") return table def demo(chat): chat.info_context = chat.rag_context_from_monster_table("aboleth") query = "What is the armor class of a Black Dragon?" chat.query_model(query) query = "Is a Purple Worm wise?" chat.query_model(query) query = "Tell me about a Quasit's agility." chat.query_model(query) query = "Is a Giant Wasp perceptive?" chat.query_model(query) query = "Is a Rakshasa it strong?" chat.query_model(query) query = "Tell me about a Raven's traits." chat.query_model(query) if __name__ == "__main__": embedding_model = SentenceTransformer("all-MiniLM-L12-v2") monster_table = init_monster_table(embedding_model) chat = Chatbot() chat.info_context = chat.rag_context_from_monster_table("aboleth") demo(chat)
Honorable Mentions
We are excited to honor the additional outstanding projects by Vikas Reddy, Vincent Müller, Alvin Lee, Vedant Tamgadge, Strikanth Thokala, Allen McCrea, Samy Gauquelin, Asaad Almutareb, Simon Schafer, and Faraz Ahmad.
Join the Intel® Liftoff Program
Are you inspired by these remarkable innovations? If you're an AI developer or startup founder looking to elevate your project, consider joining the Intel® Liftoff Program for Startups. This program provides resources, community support, and a platform to take your tech innovations to the next level.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.