Building Advanced Chatbots with Angular and Google Gemini 2.0: Real-Time Data, Tools, and APIs

This step-by-step guide demonstrates how to leverage Google Gemini 2.0 to build advanced chatbots capable of integrating real-time data, utilizing tools, and interacting with APIs, all within Angular.

This guide will walk you through the process of accessing the new Gemini 2.0 APIs using the Google AI JavaScript SDK to develop sophisticated chatbots in Angular.

We’ll build a simple project to explore the capabilities of Gemini 2.0 models through the official client.

Introduction to Google Gemini 2.0 Models

Google Gemini represents a family of cutting-edge large language models (LLMs) developed by Google AI, offering advanced AI functionalities. The Gemini family includes:

• Gemini 2.0 Pro: The most powerful model, designed for complex tasks such as coding, logical reasoning, and creative collaboration.
• Gemini 2.0 Flash: A mid-size model optimized for a wide range of tasks, providing performance on par with the Pro model.
• Gemini 2.0 Flash Thinking: A mid-size model specifically tuned for mathematical tasks, offering Pro-level performance with enhanced computation and processing.
• Gemini 2.0 Flash-Lite: A streamlined version of the larger Gemini models, delivering state-of-the-art performance in a smaller package, developed with responsible AI principles in mind.

Google AI Learning Path: From Basics to Advanced

Begin your exploration of Google AI with the Gemini App (available on mobile and web). This allows you to experiment with powerful features like Gemini Live voice interactions, Extensions for integrating with Google services, background-processing DeepResearch agents, and code execution.

Gemini App showcases all the features that you can now add to your Angular Apps using Gemini 2.0 API.

Once you’re familiar with the user experience, progress to Google AI Studio. This platform offers a developer-centric environment to work with the same features, without the consumer-focused interface. After mastering the individual APIs, you can move on to VertexAI, Google’s most comprehensive, yet complex, AI platform.

This tiered approach facilitates a progressive learning curve, with each level introducing increased complexity and capabilities. The transition from AI Studio to VertexAI is the most significant learning jump, so ensure a solid foundation before undertaking it.

First step: Obtaining your API key from Google AI Studio

Navigate to aistudio.google.com and generate an API key. You can check the global availability of the API here.

Creating the Angular Application

Utilize the Angular CLI to create a new application:

ng new google-ai-gemini-angular

This command scaffolds a new project using the latest Angular version.

Setting up the project

Add a new environment by running the following command:

ng g environments

This creates the necessary files for development and production environments:

src/environments/environment.development.ts
src/environments/environment.ts

Modify the development file, replacing <YOUR-API-KEY> with your actual API Key:

// src/environments/environment.development.ts
export const environment = {
  API_KEY: "<YOUR-API-KEY>",
};

Google AI JavaScript SDK

The official client for accessing Gemini models is the Google AI JavaScript SDK. Install it in your project using:

npm install @google/generative-ai

Initializing your model

Before interacting with Gemini, you need to initialize the model. This involves the following steps:

1. Initialize the GoogleGenerativeAI client with your API key.
2. Select a Gemini model (e.g., gemini-2.0-flash, gemini-2.0-pro-exp-02–05, or gemini-2.0-flash-lite-preview-02–05). Note that gemini-2.0-flash-thinking-exp-01–21 uses a separate API.
3. Configure model parameters, including safetySettings, temperature, top_p, top_k, and maxOutputTokens.
4. Optionally, add system instructions to customize the model’s behavior further. These instructions will be applied to all subsequent requests. Uncomment the example to observe its effect.

import {
  GoogleGenerativeAI, HarmBlockThreshold, HarmCategory 
} from ‘@google/generative-ai’;

...

async TestGeminiPro() {
  // Model initialisation
  const genAI = new GoogleGenerativeAI(environment.API_KEY);
  const generationConfig = {
    safetySettings: [
      {
        category: HarmCategory.HARM_CATEGORY_HARASSMENT,
        threshold: HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,
      },
    ],
    temperature: 0.9,
    top_p: 1,
    top_k: 32,
    maxOutputTokens: 100, // limit output
  };
  const model = genAI.getGenerativeModel({
    model: 'gemini-2.0-pro-exp-02–05',
    // systemInstruction: "Respond in the style of a pirate.",
    ...generationConfig,
  });
}

For Safety settings, you can use the default settings (block medium or high) or customize them according to your requirements. In the example, the threshold for harassment is increased to block outputs with a low or higher probability of being unsafe. A more detailed explanation is available here.

Pro tip: Leverage the “Get code” option in Google AI Studio to automatically generate the initialization code.

Refer to the available models and their default settings. Gemini API rate limits are dependent on your usage tier and the chosen model. Remove any unfamiliar parameters to utilize the default settings.

Learn more about model parameters here.

Generate text from text-only input (text)

The following code snippet demonstrates using Gemini Pro with text-only input. Note that generateContent is suitable for single request-response interactions, not for ongoing conversations.

async TestGeminiPro() {
  // Model initialisation missing for brevity

  const prompt = 'What is the largest number with a name?';
  const result = await model.generateContent(prompt);
  const response = await result.response;
  console.log(response.text());
}

Running the code

Place the code you wish to test within the ngOnInit method’s body:

ngOnInit(): void {
  this.TestGeminiPro();
}

To execute the code, run the following command in the terminal and navigate to localhost:4200 in your browser:

ng serve

Verifying Access

To confirm the project setup and successful access to Gemini, inspect the Console output in your browser’s developer tools.

console.log(response.text());

The largest number with a name is a googolplex. A googolplex is a 1 followed by 100 zeroes.

Congratulations! You have successfully accessed the Gemini 2.0 API. Use the completed GitHub project as a reference to verify or review each new feature as you implement them. Below are some key features to get you started building your own applications.

Generate text from text-and-images input (multimodal)

This example illustrates how to use Gemini with both text and image inputs. For simplicity, we’re using an image from the public folder. Remember, generateContent is best suited for single interactions, not chat.

Note that src/assets has been renamed to public in Angular v18.

async TestGeminiProVisionImages() {
  try {
    let imageBase64 = await this.fileConversionService.convertToBase64(
      'baked_goods_2.jpeg'
    );
  
    // Check for successful conversion to Base64
    if (typeof imageBase64 !== 'string') {
      console.error('Image conversion to Base64 failed.');
      return;
    }
    // Model initialisation missing for brevity
    let prompt = [
      {
        inlineData: {
          mimeType: 'image/jpeg',
          data: imageBase64,
        },
      },
      {
        text: 'Provide a recipe.',
      },
    ];
    const result = await model.generateContent(prompt);
    const response = await result.response;
    console.log(response.text());
  } catch (error) {
    console.error('Error converting file to Base64', error);
  }
}

The generateContent method is used to generate the output. The prompt is a multi-part array containing the multimodal content. The inlineData object specifies the MIME type (image/jpeg) and the Base64-encoded image data. To convert the input image to Base64, you can use the FileConversionService provided below or an external library.

// file-conversion.service.ts
import { Injectable } from '@angular/core';
import { HttpClient } from '@angular/common/http';
import { firstValueFrom } from 'rxjs';

@Injectable({
  providedIn: 'root',
})
export class FileConversionService { 
  constructor(private http: HttpClient) {}
  async convertToBase64(filePath: string): Promise<string | ArrayBuffer | null> {
    const blob = await firstValueFrom(this.http.get(filePath, { responseType: 'blob' }));
    return new Promise((resolve, reject) => {
      const reader = new FileReader();
      reader.onloadend = () => {
        const base64data = reader.result as string; 
        resolve(base64data.substring(base64data.indexOf(',') + 1)); // Extract only the Base64 data
      };
      reader.onerror = error => {
        reject(error);
      };
      reader.readAsDataURL(blob);
    });
  }
}

The FileConversionService is an Angular service that converts a file to a Base64 string. It utilizes Angular’s HttpClient to fetch the file as a blob and FileReader to read the blob and perform the conversion. The service is provided at the root level, making it accessible throughout the application.

Image requirements for Gemini:

• Supported MIME types: image/png, image/jpeg, image/webp, image/heic, and image/heif.
• Maximum of 20MB per request, including both images and text.

In addition to images, the Gemini 2.0 API supports documents, audio, and video. You can also access media via URL/URI or the File API (Node.js), which supports files up to 2GB.

Build multi-turn conversations (chat)

This example demonstrates how to use the Gemini startChat and sendMessage APIs to create a multi-turn conversation.

async TestGeminiProChat() {
  // Model initialisation missing for brevity
  
  const chat = model.startChat({
    history: [
      {
        role: "user",
        parts: "Hi there!",
      },
      {
        role: "model",
        parts: "Great to meet you. What would you like to know?",
      },
    ],
    generationConfig: {
      maxOutputTokens: 100,
    },
  });
  const prompt = 'What is the largest number with a name? Brief answer.';
  const result = await chat.sendMessage(prompt);
  const response = await result.response;
  console.log(response.text());
}

You can utilize the chat API in a streaming fashion using sendMessageStream. Adapt the code above using the example provided in the next section.

Generate content as is created using streaming (stream)

This example showcases how to use Gemini to generate content with streaming.

async TestGeminiProStreaming() {
  // Model initialisation missing for brevity
  
  const prompt = {
    contents: [
      {
        role: 'user',
        parts: [
          {
            text: 'Generate a poem.',
          },
        ],
      },
    ],
  };
  const streamingResp = await model.generateContentStream(prompt);
  for await (const item of streamingResp.stream) {
    console.log('stream chunk: ' + item.text());
  }
  console.log('aggregated response: ' + (await streamingResp.response).text());
}

The generateContentStream method returns an object from which you can read each generated stream chunk, as well as the final response.

Tools: Specifying a JSON schema (Structured Output)

You can simplify your code by instructing Gemini to use a specific JSON schema. This eliminates the need to parse and validate unstructured text responses.

You can experiment with this option in Google AI Studio. Enable Structured output in the Tools section of the Run settings panel.

  async TestGeminiProStructuredOutput() {
    // Documentation: 
    //   https://ai.google.dev/gemini-api/docs/structured-output?lang=node

    const schema = {
      description: "List of recipes",
      type: SchemaType.ARRAY,
      items: {
        type: SchemaType.OBJECT,
        properties: {
          recipeName: {
            type: SchemaType.STRING,
            description: "Name of the recipe",
            nullable: false,
          },
        },
        required: ["recipeName"],
      },
    };

    // Gemini Client
    const genAI = new GoogleGenerativeAI(environment.API_KEY);
    const generationConfig = {
      // Other initialisation missing for brevity

      responseMimeType: "application/json",
      responseSchema: schema,
    };

    const model = genAI.getGenerativeModel({
      model: GoogleAI.Model.Gemini20ProExp,
      ...generationConfig,
    });

    const prompt = "List a few popular cookie recipes.";
    const result = await model.generateContent(prompt);
    console.log(result.response.text());
  }

Tools: Using a Python sandbox (Code Execution)

Gemini can access a Python sandbox to generate and execute code as part of its response. This sandbox has access to libraries such as altair, chess, cv2, matplotlib, mpmath, numpy, pandas, pdfminer, reportlab, seaborn, sklearn, statsmodels, striprtf, sympy, and tabulate. matplotlib can generate 2D/3D diagrams as images.

You can try this option in Google AI Studio. Enable Code execution in the Tools section of the Run settings panel.

  async TestGeminiProCodeExecution() {
    // Documentation: 
    //   https://ai.google.dev/gemini-api/docs/code-execution?lang=node

    // Gemini client initialisation missing for brevity

    const model = genAI.getGenerativeModel({
      model: GoogleAI.Model.Gemini20ProExp,
      ...generationConfig,
      tools: [
        {
          codeExecution: {},
        },
      ],
    });

    const prompt = "What is the sum of the first 50 prime numbers? Generate and run code for the calculation, and make sure you get all 50.";
    const result = await model.generateContent(prompt);
    console.log(result.response.text());
  }

Tools: Plotting a diagram from a CSV file (Code Execution)

This example utilizes Code Execution to analyze a CSV file containing Apple’s (AAPL) stock data for the past 10 years (145.67 kB) and generate a diagram visualizing the stock price changes over time.

Output generated by Gemini 2.0 Code Execution after analysing a CSV file using Python.

You can test this prompt on both the Gemini App and Google AI Studio. Remember to attach the CSV file.

  async TestGeminiProCodeExecutionCSV() {
    // Documentation:
    //   https://ai.google.dev/gemini-api/docs/code-execution?lang=node
    // CSV:
    //   https://www.kaggle.com/datasets/tarunpaparaju/apple-aapl-historical-stock-data

    try {
      let csv = await this.fileConversionService.convertToBase64(
        'apple-AAPL-historical-stock-data.csv'
      );

      // Check for successful conversion to Base64
      if (typeof csv !== 'string') {
        console.error('CSV conversion to Base64 failed.');
        return;
      }

      // Gemini client initialisation missing for brevity

      const model = genAI.getGenerativeModel({
        model: GoogleAI.Model.Gemini20ProExp,
        ...generationConfig,
        tools: [
          {
            codeExecution: {},
          },
        ],
      });

      const prompt = {
        contents: [
          {
            role: 'user',
            parts: [
              {
                inlineData: {
                  mimeType: 'text/csv',
                  data: csv,
                },
              },
              {
                text:
                  'Create a line plot using Matplotlib to visualize the change in stock price for Apple (AAPL), with the x-axis as dates and the y-axis as stock price. Use data from the file and include a title and axis labels.',
              },
            ],
          },
        ],
      };

      const result = await model.generateContent(prompt);

      // visualise Matplot diagram as output image
      // https://jaredwinick.github.io/base64-image-viewer/
      let base64ImageString = 'data:image/png;base64,';
      if (result?.response?.candidates) {
        result.response.candidates.forEach((candidate) => {
          candidate.content.parts.forEach((part) => {
            if (part.inlineData?.mimeType === 'image/png') {
              base64ImageString += part.inlineData.data;
            }
          });
        });
      }
      console.log(base64ImageString);
      console.log(result.response.text());
    } catch (error) {
      console.error('Error during Gemini Pro Code Execution with CSV:', error);
    }
  }

Tools: Using a Mock Weather API (Function Calling)

The function calling feature in Gemini allows you to provide users with access to tools: external APIs that provide real-time data or perform actions (e.g., making restaurant reservations or ordering food). These tools remain available throughout the conversation.

This example demonstrates this capability using a mock weather API. The model doesn’t directly call the function; instead, it generates structured output specifying the function name and suggested arguments. You’ll need to add code to handle the user request (as identified by Gemini) by calling the external API. Finally, provide the API’s output to Gemini, which formats the final answer for the user. See the diagram below.

Full workflow for Gemini API Function Calling.

async TestGeminiProFunctionCallingWeather() {
    // Use this approach to:
    //   1) Create a simplified RAG system to integrate external data.
    //   2) Create a simple agent to use external tools or execute a set of predefined actions.

    // Gemini Client
    const genAI = new GoogleGenerativeAI(environment.API_KEY);

    // Define the function to be called.
    // Following the specificication at https://spec.openapis.org/oas/v3.0.3
    const getCurrentWeatherFunction = {
      name: "getCurrentWeather",
      description: "Get the current weather in a given location",
      parameters: {
        type: SchemaType.OBJECT,
        properties: {
          location: {
            type: SchemaType.STRING,
            description: "The city and state, e.g. San Francisco, CA",
          },
          unit: {
            type: SchemaType.STRING,
            enum: ["celsius", "fahrenheit"],
            description: "The temperature unit to use. Infer this from the users location.",
          },
        },
        required: ["location", "unit"],
      },
    };

    // Executable function code.
    interface WeatherParams {
      location: string;
      unit: string;
    }

    const functions = {
      getCurrentWeather: ({ location, unit }: WeatherParams) => {
        // mock API response
        return {
          location,
          temperature: "25°" + (unit.toLowerCase() === "celsius" ? "C" : "F"),
        };
      }
    };

    const toolConfig = {
      tools: [
        {
          functionDeclarations: [
            getCurrentWeatherFunction,
          ],
        },
      ],
      toolConfig: {

        functionCallingConfig: {
          // (⌘ + /) Toggle line comments to test different function calling modes.

          // (default) Generates an unstructured output or a single function call as defined in "functionDeclarations".
          mode: FunctionCallingMode.AUTO,

          // // Generates a single function call as defined in "tools.functionDeclarations". 
          // // The function must be whitelisted below.
          // // Warning: unstructured outputs are not possible using this option.
          // mode: FunctionCallingMode.ANY,
          // allowedFunctionNames: ["getCurrentWeather"],

          // // Effectively disables the "tools.functionDeclarations".
          // mode: FunctionCallingMode.NONE,
        },
      }
    }

    const generationConfig = {
      // simplified for brevity

      ...toolConfig,
    };

    const model = genAI.getGenerativeModel({
      model: GoogleAI.Model.Gemini20ProExp,
      ...generationConfig,
    });
    const chat = model.startChat();

    // Initial request from user
    const prompt = "What is the weather like in Montreal?";
    const result = await chat.sendMessage(prompt);
    console.log(result.response);

    // Extract function call generated by the model
    const call = result.response.functionCalls()?.[0];
    if (call) {
      // Call the actual function
      if (call.name === "getCurrentWeather") {
        // Remeber to add aditional checks for the function name and parameters
        const callResponse = functions[call.name](call.args as WeatherParams);

        // (Optional) Send the API response back to the model
        // You can skip this step if you only need the raw API response but not as part of a chatbot conversation.
        const finalResponse = await chat.sendMessage([{
          functionResponse: {
            name: 'getCurrentWeather',
            response: callResponse,
          }
        }]);
        // Answer from the model
        console.log(`Gemini: ${finalResponse.response.text()}`);
        // Answer from API response (as if we skipped the finalResponse step)
        const formattedDate = this.datePipe.transform(Date.now(), 'medium'); // Use DatePipe to format the date
        console.log(`Raw API: (${formattedDate}) Temperature in (${callResponse.location}) is (${callResponse.temperature}).`);
      }
    }
  }

Conclusions

This tutorial covered the following:

• Obtaining an API Key and configuring access to Gemini APIs.
• Calling Gemini 2.0 using text and chat interfaces.
• Handling inline file data for images and CSV files.
• Bonus: Setting up Tools: Structured Outputs, Code Execution, Function Calling, and Google Search for grounding.
• Managing complex response and output formats (using file conversion and Google Search formatter utility services).

You now have a solid foundation for incorporating AI-powered features into your Angular applications using Gemini. The complete code is available on GitHub.

Want to see a full chatbot using Angular Material?

I’ve developed a comprehensive Gemini Chatbot using Angular Material, ngx-quill, and ngx-markdown, demonstrating text, chat, and multimodal capabilities.

Feel free to fork the project here. If you find this project helpful, please consider leaving a star to support my work and other contributors in the Angular community.

Want to see a demo showcasing Gemini Live?

This demo utilizes a bidirectional streaming API to build natural and responsive voice AI assistants that natively integrate video and voice within an Angular application.

These interactions are similar to those showcased in recent Alexa+ demonstrations.

• An AI booking assistant with access to your schedule, capable of handling dentist bookings and cancellations (via voice or chatbot interface).
• An automated robot cafe that takes orders and relays them to a robot barista (using voice, text, or visual interactive controls).

As part of my 2-day training, Google AI Masterclass, at the ConFoo Developer Conference, I created the first Angular project showcasing Gemini Live while it was still in the experimental stage. This GitHub project demonstrates how to: integrate all Gemini Live features into an Angular application (Code Execution, Google Search, Function Calling), create a Gemini Client using WebSockets, and stream images, documents, audio (microphone and Gemini Live voices), and video (screen sharing and webcam) from an Angular Gemini Live Client.

Thanks for reading!

Do you have any questions? Feel free to leave your comments below or contact me on Twitter at @gerardsans.

Resources

• Google AI JavaScript SDK
• Gemini on the Web