Welcome to our exploration of Challenge 5 in the Advent of GenAI Hackathon, where the focus shifts to demystifying Python code. Building on the momentum of Challenge 4, this event continued to harness the power of Generative AI and Large Language Models (LLMs) on the Intel Developer Cloud (IDC), leveraging Intel GPUs. This challenge, aimed at AI developers, startup founders, and students, pushed the envelope in AI technology, demonstrating the practical applications of advanced AI tools in programming.
Challenge 5: Solve Python Code Mysteries With Explanations
In the fifth challenge of the Advent of GenAI Hackathon, participants were tasked with developing a Python-based tool to simplify Python code. The instructions were to create a tool that could use Retrieval-Augmented Generation (RAG) combined with a vector database like LanceDB, along with Prediction Guard's Large Language Model (LLM) APIs, such as WizardCoder or local models like CodeLLama on the Intel Developer Cloud (IDC).
The main objective was to build a solution that generates concise, understandable explanations for Python code snippets. The tool would need to use open-source documentation and datasets to provide accurate and user-friendly insights into the functionality of the code, prioritizing clarity and ease of understanding.
For users, tools would need to feature an interface (e.g., an input box in a Jupyter Notebook) where users can submit Python code and natural language questions, receiving precise answers to their queries.
Technical Expectations:
- Understanding RAG's Application: Learn how RAG's retrieval component can fetch relevant documentation or examples from the vector database, while its generation component formulates clear explanations.
- Choosing a Vector Database: Opt for a database rich in Python documentation, code snippets, and explanations, including resources like Stack Overflow, GitHub repositories, or official Python documentation.
- Model Integration: Employ the Prediction Guard LLM API with models like WizardCoder or use local models such as CodeLlama available on IDC, focused on programming-related content.
User-Friendly Design: Ensure the tool is intuitive, capable of accepting various code snippet formats, and provides explanations in simple, understandable language.
Technical Setup:
- Database and Model Integration: Select a suitable vector database, enrich it with extensive Python documentation, and integrate a model using Prediction Guard’s LLM API, such as WizardCoder, or utilize a local model like CodeLlama. The RAG should effectively link code snippets with relevant and precise documentation and explanations.
The Top Submissions of Challenge
Drumroll, please, for the standout winners of Challenge 5: these submissions were a testament to the ingenuity and mastery in AI-driven code explanation by the participants. Despite being the most challenging, the participants exceeded expectations with their innovative approaches. The following are the top 4 submissions that showcased remarkable skill in explaining complex Python code through AI.
Submission #1: "Intelligent Agent for Library Selection and Code Generation" by Tomáš Barczi
Tomáš Barczi's project took a novel approach, focusing on helping users implement functionality rather than just explaining code. His solution identified the most suitable Python libraries for tasks, then provided and explained the relevant code. The process involved cloning repositories, parsing READMEs (as shown in Image 1), and utilizing CodeLlama for library descriptions. These were stored in a vector database. For retrieval, the agent selected the best library match and detailed functions with explanations, including documentation, GitHub links, and file locations (illustrated in Images 2 and 3). Barczi's emphasis was on aiding non-programmers and beginners in coding, believing this would have the most significant impact. His solution represented a shift from simply explaining code to enabling users to write it efficiently.
Final Output:
Tomáš Barczi, in his review of the fifth challenge, emphasized the innovative use of LLM (Large Language Models) in programming. He noted, "For the challenge 5 the task was to use LLM models for code understanding and explanation." Barczi aimed to diverge from the typical application of LLMs in explaining code snippets, stating, "I wanted to make something different." He believed the most significant impact of LLMs lies not in aiding experienced programmers, but in assisting "non-programmers or beginner programmers write anything."
Barczi's chosen use case involved a system where users describe their desired functionality. He used the example of splitting documents into chunks, necessary for a RAG (Retrieve and Generate) system. He critiqued the standard RAG approach for potentially causing a mess with dependencies and imports. Instead, he proposed a "multiple step approach" where the system first identifies the best library for a task before providing the relevant code. This approach, according to Barczi, encourages finding and reusing the best tool for a given task.
Submission #2: "RAG vs Non-RAG Comparison and Custom Model Implementation" by Simon Hanly-Jones, Emmanuel Isaac, and Ryan Mann
The team of Simon Hanly-Jones, Emmanuel Isaac, and Ryan Mann developed a unique comparison between RAG and non-RAG models. Their project involved creating an API and a detailed implementation document. The team expressed their enjoyment and learning experience from the challenge and the hackathon. They noted the difficulty in enhancing performance with RAG injection over the 'out of the box' performance achieved with CodeWizard. Their approach included scraping the official Python documentation and utilizing the "all-MiniLM-L12-v2" transformer for data storage and accessibility. The team's efforts led to a valuable exploration of different modeling techniques in AI-driven code explanation.
The non rag model used the following prompt template:
Our rag model uses this prompt template:
The key to good performance was good prompting. Using “Context” or “Informational Context” confused the model, and seemed to make it think the user was asking about the python documentation instead of the query. Reformatting the prompt to explicitly refer to the RAG injection as “Python Documentation” made it possible to obtain a meaningful contribution from the RAG injection.
The team wrapped their models in a fast API and relied on the automatically generated swagger documentation as a User Interface as required. On running the code it is available at “127.0.0.1:8000/docs” URL. This was done because fast API is a useful method to deploy LLM’s and abstract them from the end user.
To run the project, first download all files and put them in the same directory, run get_lance_db.py, which will transform the "pychunks.json" file into the vectorized database. Then run wiz_code.py, which will generate the demo_output to sdout then launch the fast API.
As Ryan Mann explained, this submission demonstrates that a “specialized model like CodeWizard can be further refined through targeted prompting, without the need for additional context injection.” In this project, his team successfully developed a chatbot capable of explaining Python code. This was achieved using a straightforward prompt template:
Their findings also point to the possibility of tailoring a model for even more specific requirements. This could be done through context injection using the Retrieval-Augmented Generation (RAG) technique. In this instance, they enhanced the chatbot's output by integrating Python documentation. This allowed the chatbot to weave the documentation into its responses. However, he also emphasized the need for caution. “Superfluous or improperly retrieved context can compromise the quality of the response, especially if it's not pertinent to the use case.”
As he put it, the key to success with this model lay in effective prompting. “The RAG component only showed improvement when the prompts were meticulously crafted to indicate the inclusion of Python documentation. Without this careful prompt engineering, the RAG's performance was significantly hindered.”
The advanced prompt they used was as follows:
The project used Python Documentation essentially as a heading and then referred to it in the instructions. This was key to the model’s performance.
Submission #3: "Comprehensive Code Explanation Tool" by Vikas Reddy B
Vikas Reddy B’s submission stood out for its ability to explain code from start to finish. The tool provided clear explanations of code functionality and offered output demonstrations, suggested improvements, and highlighted errors. This submission was notable for its practical approach, tweaking parameters and experimenting with various prompt trials to refine the output. Reddy B's tool represents a significant advancement in AI-assisted code understanding, delivering a comprehensive solution that guides users through the entire process of code analysis and optimization.
Submission #4: "Advanced Code Explanation and Optimization" by Srikanth Thokala, Pallavi Kumari, Subramanian Mohan, and Thejesh Reddy T R
This team developed a comprehensive solution using PG's WizardCoder model and Python documentation (PDFs from Python's official documentation). They enhanced the chat interface from Challenge 4, enabling it to explain, find, correct, optimize code, predict output, and generate code. A key innovation was allowing follow-up queries without re-entering the code block, using the 'earlier' keyword for code retrieval. Srikanth Thokala shared their systematic approach to the challenge, emphasizing extensive experimentation with WizardCoder and the addition of a code caching feature for improved interaction. Screenshots of Challenge 5 are attached below.
Srikanth Thokala, leading a team comprising Pallavi Kumari, Subramanian Mohan, and Thejesh Reddy T R, shared insights into their approach to the Day 5 challenge, which they found more demanding than previous tasks. Thokala stated, "The Day 5 problem statement is more challenging for my team compared to earlier problems." Despite only receiving Special recognitions for their first four challenges, they aimed to excel in this one by adding significant value. Thokala detailed their methodical approach, which involved thoroughly reviewing resources provided by Daniel and various articles. They extensively tested Prediction Guard's WizardCoder, experimenting with different prompts and analyzing outcomes.
Thokala emphasized the innovation they brought to the process, saying, "On top of that, to bring a value-add, we cached the code block so that the code interpreter could respond to the follow-up queries with no need to prompt the code block." Their rigorous testing and refinement included manual syntax error injections and using less optimized code to ensure consistent output. Concluding, Thokala expressed satisfaction with the team's enhanced skills and anticipation for the final challenge, stating, "To conclude, Day 5 surely upskilled our knowledge, and we are very much looking forward to the final day challenge."
Honorable Mentions
Several other creative submissions deserve special recognition. Shoutout to Johannes Widter, Chan Jong Na, Samy Gauquelin, Asaad Almutareb, Alvin Lee Neam Heng, and Faraz Ahmad.
Join the Intel® Liftoff Program
If you're an AI developer or startup founder, apply to join the Intel® Liftoff Program for Startups: the springboard your project needs. It’s a free program combining resources, community support, and a platform for tech innovators. Get in touch to learn more today.
I'm a proud team member of the Intel® Liftoff for Startups, an innovative, free virtual program dedicated to accelerating the growth of early-stage AI startups.