examples/notebooks/beam-ml/milvus_enrichment_transform.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "47053bac",
   "metadata": {},
   "outputs": [],
   "source": [
    "# @title ###### Licensed to the Apache Software Foundation (ASF), Version 2.0 (the \"License\")\n",
    "\n",
    "# Licensed to the Apache Software Foundation (ASF) under one\n",
    "# or more contributor license agreements. See the NOTICE file\n",
    "# distributed with this work for additional information\n",
    "# regarding copyright ownership. The ASF licenses this file\n",
    "# to you under the Apache License, Version 2.0 (the\n",
    "# \"License\"); you may not use this file except in compliance\n",
    "# with the License. You may obtain a copy of the License at\n",
    "#\n",
    "#   http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing,\n",
    "# software distributed under the License is distributed on an\n",
    "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
    "# KIND, either express or implied. See the License for the\n",
    "# specific language governing permissions and limitations\n",
    "# under the License"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa881240-2f38-4335-9d4d-444776d77c92",
   "metadata": {},
   "source": [
    "# Use Apache Beam and Milvus to enrich data\n",
    "\n",
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/beam-ml/milvus_enrichment_transform.ipynb\"><img src=\"https://raw.githubusercontent.com/google/or-tools/main/tools/colab_32px.png\" />Run in Google Colab</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/milvus_enrichment_transform.ipynb\"><img src=\"https://raw.githubusercontent.com/google/or-tools/main/tools/github_32px.png\" />View source on GitHub</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0611da21-d031-4b16-8301-9b76bda731e7",
   "metadata": {},
   "source": [
    "This notebook shows how to enrich data by using the Apache Beam [enrichment transform](https://beam.apache.org/documentation/transforms/python/elementwise/enrichment-milvus) with [Milvus](https://milvus.io/). The enrichment transform is an Apache Beam turnkey transform that lets you enrich data by using a key-value lookup. This transform has the following features:\n",
    "\n",
    "- The transform has a built-in Apache Beam handler that interacts with Milvus data during enrichment.\n",
    "- The enrichment transform uses client-side throttling to rate limit the requests. The default retry strategy uses exponential backoff. You can configure rate limiting to suit your use case.\n",
    "\n",
    "This notebook demonstrates the following search engine optimization use case:\n",
    "\n",
    "A specialized technical search engine company wants to improve its query result relevance by dynamically enriching search results with semantically related content. The example uses a vector database of technical articles and documentation stored in Milvus to enrich incoming user queries. The enriched data is then used to provide users with more comprehensive and contextually relevant search results, especially for complex technical topics.\n",
    "\n",
    "## Before you begin\n",
    "Set up your environment and download dependencies.\n",
    "\n",
    "### Install Apache Beam\n",
    "To use the enrichment transform with the built-in Milvus handler, install the Apache Beam SDK version 2.67.0 or later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e550cd55-e91e-4d43-b1bd-b0e89bb8cbd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Disable tokenizers parallelism to prevent deadlocks when forking processes\n",
    "# This avoids the \"huggingface/tokenizers: The current process just got forked\" warning.\n",
    "import os\n",
    "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "31747c45-107a-49be-8885-5a6cc9dc1236",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "# The Apache Beam test dependencies are included here for the TestContainers\n",
    "# Milvus standalone DB container that will be used later in the demo.\n",
    "!pip install rich sentence_transformers llama_index --quiet\n",
    "!pip install apache_beam[milvus,gcp,test,interactive] --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "666e0c2b-0341-4b0e-8d73-561abc39bb10",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/dev/beam/sdks/python/.venv/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py:2249: UnsupportedFieldAttributeWarning: The 'validate_default' attribute with value True was provided to the `Field()` function, which has no effect in the context it was used. 'validate_default' is field-specific metadata, and can only be attached to a model field using `Annotated` metadata or by assignment. This may have happened because an `Annotated` type alias using the `type` statement was used, or if the `Field()` function was attached to a single member of a union type.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "# Standard library imports\n",
    "from collections import defaultdict\n",
    "from dataclasses import asdict\n",
    "from math import ceil\n",
    "from typing import Any, Dict, List\n",
    "import tempfile\n",
    "import uuid\n",
    "import shutil\n",
    "\n",
    "# Third-party imports\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from pymilvus import (\n",
    "    DataType, \n",
    "    CollectionSchema, \n",
    "    FieldSchema, \n",
    "    Function, \n",
    "    FunctionType, \n",
    "    MilvusClient, \n",
    "    RRFRanker\n",
    ")\n",
    "from pymilvus.milvus_client import IndexParams\n",
    "from rich import print_json\n",
    "from sentence_transformers import SentenceTransformer\n",
    "from torch import cuda\n",
    "from llama_index.core.text_splitter import SentenceSplitter\n",
    "\n",
    "# Apache Beam imports\n",
    "import apache_beam as beam\n",
    "from apache_beam.ml.rag.types import Chunk, Content, Embedding\n",
    "from apache_beam.ml.rag.chunking.base import ChunkingTransformProvider\n",
    "from apache_beam.ml.rag.embeddings.huggingface import HuggingfaceTextEmbeddings\n",
    "from apache_beam.ml.rag.enrichment.milvus_search_it_test import MilvusEnrichmentTestHelper\n",
    "from apache_beam.ml.rag.enrichment.milvus_search import (\n",
    "    HybridSearchParameters, \n",
    "    KeywordSearchMetrics, \n",
    "    KeywordSearchParameters,\n",
    "    MilvusCollectionLoadParameters, \n",
    "    MilvusConnectionParameters, \n",
    "    MilvusSearchEnrichmentHandler,\n",
    "    MilvusSearchParameters, \n",
    "    SearchStrategy, \n",
    "    VectorSearchMetrics, \n",
    "    VectorSearchParameters\n",
    ")\n",
    "from apache_beam.ml.transforms.base import MLTransform\n",
    "from apache_beam.ml.transforms.embeddings import huggingface\n",
    "from apache_beam.runners.interactive import interactive_beam as ib\n",
    "from apache_beam.transforms.enrichment import Enrichment"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "338808ff-3f80-48e5-9c76-b8d19f8769b7",
   "metadata": {},
   "source": [
    "## Collect Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d83ad549-5ee1-4a4c-ae5a-e638c3d0279f",
   "metadata": {},
   "source": [
    "This content has been paraphrased from publicly available information on the internet using a large language model (OpenAI’s GPT-4) and is provided for informational purposes only."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d39a070a-206d-41f6-9033-fff0d5ea2128",
   "metadata": {},
   "source": [
    "The third data point, related to Google Beam, was intentionally included to illustrate the importance of metadata filtering (filtered search) in Milvus—such as when a user searches for the term “Beam.” without it the vector database retrieval engine may confuse between Apache Beam and Google Beam."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "38781cf5-e18f-40f5-827e-2d441ae7d2fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = [\n",
    "  {\n",
    "    \"id\": \"1\",\n",
    "    \"title\": \"Apache Beam: Unified Model for Batch and Streaming Data\",\n",
    "    \"keywords\": [\"Apache Beam\", \"stream processing\", \"batch processing\", \"data pipelines\", \"SDK\"],\n",
    "    \"tags\": [\"Data Engineering\", \"Open Source\", \"Streaming\", \"Batch\", \"Big Data\"],\n",
    "    \"content\": (\n",
    "      \"Apache Beam is an open-source framework that provides a consistent programming model for both batch and streaming data processing. \"\n",
    "      \"Developed originally by Google, it allows developers to write pipelines that can run on multiple engines, such as Apache Flink, Spark, and Google Cloud Dataflow. \"\n",
    "      \"Beam uses abstractions like PCollections (data containers) and PTransforms (operations) to define the flow of data. \"\n",
    "      \"The framework promotes portability through its runner architecture, letting the same pipeline execute on different backends. \"\n",
    "      \"Support for multiple SDKs, including Java and Python, makes it accessible for a broad audience. \"\n",
    "      \"Key features include support for event time, windowing, triggers, and stateful processing, which are essential for handling real-time data effectively. \"\n",
    "      \"Beam is ideal for building ETL jobs, real-time analytics, and machine learning data pipelines. \"\n",
    "      \"It helps teams focus on logic rather than infrastructure, offering flexibility and scalability in handling unbounded and bounded data sources. \"\n",
    "      \"Apache Beam also supports a wide range of connectors for both input and output, including Kafka, BigQuery, and JDBC-based systems. \"\n",
    "      \"This makes it easy to integrate Beam into existing data ecosystems. Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing. \"\n",
    "      \"The concept of runners enables developers to write once and run anywhere, which is particularly appealing for organizations that want to avoid vendor lock-in. \"\n",
    "      \"The Beam model is based on a unified programming model that decouples pipeline logic from execution. \"\n",
    "      \"This makes it easier to reason about time and state in both batch and streaming pipelines. \"\n",
    "      \"Advanced features like late data handling, watermarks, and session windowing allow for more accurate and meaningful processing of real-world data. \"\n",
    "      \"Beam also integrates with orchestration tools and monitoring systems, allowing for production-grade deployments. \"\n",
    "      \"Community support and contributions have grown significantly, making Beam a stable and evolving ecosystem. \"\n",
    "      \"Many cloud providers offer native support for Beam pipelines, and it's increasingly a core component in modern data platform architectures.\"\n",
    "    )\n",
    "  },\n",
    "  {\n",
    "    \"id\": \"2\",\n",
    "    \"title\": \"Google Cloud Dataflow: Run Apache Beam in the Cloud\",\n",
    "    \"keywords\": [\"Google Cloud\", \"Dataflow\", \"Apache Beam\", \"serverless\", \"stream and batch\"],\n",
    "    \"tags\": [\"Cloud Computing\", \"Data Pipelines\", \"Google Cloud\", \"Serverless\", \"Enterprise\"],\n",
    "    \"content\": (\n",
    "      \"Google Cloud Dataflow is a fully managed service that runs Apache Beam pipelines in the cloud. \"\n",
    "      \"It abstracts away infrastructure management and handles dynamic scaling, load balancing, and fault tolerance. \"\n",
    "      \"Developers can focus on writing data logic using the Beam SDK and deploy it easily to Google Cloud. \"\n",
    "      \"Dataflow supports both batch and stream processing and integrates seamlessly with other Google services like BigQuery, Pub/Sub, and Cloud Storage. \"\n",
    "      \"Its autoscaling capabilities allow it to adapt to changing data volumes, optimizing for cost and performance. \"\n",
    "      \"Features like monitoring dashboards, job templates, and built-in logging make it suitable for both development and production use. \"\n",
    "      \"With support for event time processing, stateful functions, and windowing, Dataflow is well-suited for real-time analytics and data transformation tasks. \"\n",
    "      \"It’s a key component for architects building scalable, cloud-native data platforms. \"\n",
    "      \"Dataflow also offers templates for common ETL tasks, helping teams get started quickly with minimal setup. \"\n",
    "      \"Its integration with Cloud Functions and Cloud Composer enables event-driven and orchestrated workflows. \"\n",
    "      \"Security and compliance are built-in with IAM roles, encryption at rest and in transit, and audit logging, making it suitable for enterprise environments. \"\n",
    "      \"For developers, Dataflow provides local testing capabilities and a unified logging system through Cloud Logging. \"\n",
    "      \"It also supports SQL-based pipeline definitions using BigQuery, which lowers the barrier to entry for analysts and data engineers. \"\n",
    "      \"Dataflow’s streaming engine significantly improves performance and reduces costs by decoupling compute and state management. \"\n",
    "      \"In summary, Google Cloud Dataflow not only simplifies the deployment of Apache Beam pipelines but also enhances them with cloud-native features. \"\n",
    "      \"Its managed runtime, high availability, and integration with the broader Google Cloud ecosystem make it a powerful tool for modern data processing.\"\n",
    "    )\n",
    "  },\n",
    "  {\n",
    "    \"id\": \"3\",\n",
    "    \"title\": \"Google Beam: 3D Communication Powered by AI\",\n",
    "    \"keywords\": [\"Google Beam\", \"Project Starline\", \"3D video\", \"AI communication\", \"real-time meetings\"],\n",
    "    \"tags\": [\"AI\", \"Communication\", \"3D Technology\", \"Remote Work\", \"Enterprise Tech\"],\n",
    "    \"content\": (\n",
    "      \"Google Beam is an innovative video communication platform that builds on the research of Project Starline. It uses AI, 3D imaging, and light field rendering to create immersive, lifelike video calls. \"\n",
    "      \"Designed to replicate in-person interaction, Beam allows users to see life-sized, three-dimensional representations of each other without the need for headsets. \"\n",
    "      \"This breakthrough makes remote conversations feel natural—capturing facial expressions, eye contact, and subtle gestures that traditional video conferencing often misses. \"\n",
    "      \"Beam reduces meeting fatigue and enhances engagement, making it ideal for enterprise collaboration, interviews, and virtual presence scenarios. \"\n",
    "      \"Powered by Google AI, Beam represents a significant leap in communication technology. \"\n",
    "      \"Major companies like Salesforce, Deloitte, and NEC are already exploring its impact on digital collaboration. \"\n",
    "      \"Google is partnering with HP to build and distribute Beam hardware, designed to work with existing productivity and video tools. \"\n",
    "      \"Currently in limited early access for enterprise partners, Google Beam aims to redefine virtual meetings by bridging the gap between digital and physical presence. \"\n",
    "      \"It’s a promising step toward more human and effective remote interactions.\"\n",
    "    )\n",
    "  }\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "758c2af7-12c7-477b-9257-3c88712960e7",
   "metadata": {},
   "source": [
    "## Exploratory Data Analysis (EDA)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e751905-7217-4571-bc07-991ef850a6b2",
   "metadata": {},
   "source": [
    "### Average Words/Tokens per Doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "489e93b6-de41-4ec3-be33-a15c3cba12e8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th># Words</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>3.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>253.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>72.858310</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>172.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>224.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>277.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>294.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>312.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          # Words\n",
       "count    3.000000\n",
       "mean   253.666667\n",
       "std     72.858310\n",
       "min    172.000000\n",
       "25%    224.500000\n",
       "50%    277.000000\n",
       "75%    294.500000\n",
       "max    312.000000"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The second video may skew the average tokens results since it is a youtube short video.\n",
    "contents = [c['content'] for c in corpus]\n",
    "content_lengths = [len(content.split(\" \")) for content in contents]\n",
    "df = pd.DataFrame(content_lengths, columns=['# Words'])\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "eb32aad0-febd-45af-b4bd-e2176b07e2dc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The mean word count for each video is about 254 words, which corresponds to a rough token count of 331 tokens.\n"
     ]
    }
   ],
   "source": [
    "mean_word_count = ceil(np.mean(content_lengths))\n",
    "token_to_word_ratio = 1.3\n",
    "approx_token_count = ceil(mean_word_count * token_to_word_ratio)\n",
    "print(f'The mean word count for each video is about {mean_word_count} words, which corresponds to a rough token count of {approx_token_count} tokens.')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "765115e1-4327-44f6-9dff-5d79121eeb02",
   "metadata": {},
   "source": [
    "## Milvus Sink I/O"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "492adeba-c6cd-404d-9d48-dfcaeca503c2",
   "metadata": {},
   "source": [
    "This could be delegated to the Beam Milvus Sink I/O once it is implemented. For now, we will use pymilvs client directly for indexing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3889aaa4-3c0c-4d71-bad3-b196b5eac8dc",
   "metadata": {},
   "source": [
    "### Setup Milvus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5ae9bc82-9ad7-46dd-b254-19cbdcdd0e07",
   "metadata": {},
   "outputs": [],
   "source": [
    "db = None\n",
    "milvus_version = \"milvusdb/milvus:v2.5.10\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "aff7b261-3330-4fa9-9a54-3fd87b42521f",
   "metadata": {},
   "outputs": [],
   "source": [
    "if db:\n",
    "    # Stop existing Milvus DB container to prevent duplicates.\n",
    "    MilvusEnrichmentTestHelper.stop_db_container(db)\n",
    "db = MilvusEnrichmentTestHelper.start_db_container(milvus_version)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "31496ee0-75a2-48ad-954e-9c4ae5abbf5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "milvus_connection_parameters = MilvusConnectionParameters(uri=db.uri, user=db.user, password=db.password, db_id=db.id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "82627714-2425-4058-9b47-d262f015caf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = MilvusClient(**milvus_connection_parameters.__dict__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e8a85f51-5d5f-4533-bf0f-ec825e613dc2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.5.10'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.get_server_version()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2344abb9-c170-4496-993e-736e2b50c2bb",
   "metadata": {},
   "source": [
    "### Define Vector Schema and Indices"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31130864-a7c6-45af-bc15-8b64bb9ff8fa",
   "metadata": {},
   "source": [
    "#### Define Fields"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e3847821-069c-412f-8c20-2406bcac1e55",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choosing `sentence-transformers/all-MiniLM-L6-v2` as our embedding generator here. It gives\n",
    "# a good balance between embedding generation speed, accuracy, and being free to use.\n",
    "embedding_model_config = {\n",
    "    \"name\": 'sentence-transformers/all-MiniLM-L6-v2',\n",
    "    \"token_limit\": 384\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c014af94-1bb7-44e4-842c-1039f4a2a11d",
   "metadata": {},
   "outputs": [],
   "source": [
    "fields = [\n",
    "    FieldSchema(name=\"id\", dtype=DataType.VARCHAR, is_primary=True, max_length=100),\n",
    "    FieldSchema(name=\"content\", dtype=DataType.VARCHAR, max_length=65279),\n",
    "    FieldSchema(name=\"embedding\", dtype=DataType.FLOAT_VECTOR, dim=embedding_model_config[\"token_limit\"]),\n",
    "    FieldSchema(name=\"sparse_embedding\", dtype=DataType.SPARSE_FLOAT_VECTOR),\n",
    "    FieldSchema(name=\"metadata\", dtype=DataType.JSON),\n",
    "    FieldSchema(name=\"title_and_content\", dtype=DataType.VARCHAR, max_length=65279+256, enable_analyzer=True),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76535a60-87f5-48e0-9c73-38aa2c6b4d0e",
   "metadata": {},
   "source": [
    "#### Define Functions for Processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "54fb3428-b007-4804-9d79-b3933d3256c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "bm25_function = Function(\n",
    "    name=\"content_bm25_emb\",\n",
    "    input_field_names=[\"title_and_content\"],\n",
    "    output_field_names=[\"sparse_embedding\"],\n",
    "    function_type=FunctionType.BM25)\n",
    "\n",
    "functions = [bm25_function]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4c2f123a-5949-4974-af48-a5db5b168c11",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'auto_id': False, 'description': '', 'fields': [{'name': 'id', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 100}, 'is_primary': True, 'auto_id': False}, {'name': 'content', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 65279}}, {'name': 'embedding', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 384}}, {'name': 'sparse_embedding', 'description': '', 'type': <DataType.SPARSE_FLOAT_VECTOR: 104>, 'is_function_output': True}, {'name': 'metadata', 'description': '', 'type': <DataType.JSON: 23>}, {'name': 'title_and_content', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 65535, 'enable_analyzer': True}}], 'enable_dynamic_field': False, 'functions': [{'name': 'content_bm25_emb', 'description': '', 'type': <FunctionType.BM25: 1>, 'input_field_names': ['title_and_content'], 'output_field_names': ['sparse_embedding'], 'params': {}}]}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema = CollectionSchema(fields=fields,functions=functions)\n",
    "schema"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04f15d4b-1192-464b-9635-cb4cbc530431",
   "metadata": {},
   "source": [
    "#### Define Indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "671f4352-2086-4428-83be-0de48926682d",
   "metadata": {},
   "outputs": [],
   "source": [
    "index_params = IndexParams()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "378909d0-3aa8-46a5-8983-3ab29a1b0049",
   "metadata": {},
   "source": [
    "#### Define Dense Vector Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "aa8baae5-7c38-4e78-ace4-304c7dc6b127",
   "metadata": {},
   "outputs": [],
   "source": [
    "index_params.add_index(\n",
    "    field_name=\"embedding\",\n",
    "    index_name=\"dense_embedding_ivf_flat\",\n",
    "    index_type=\"IVF_FLAT\",\n",
    "    metric_type=VectorSearchMetrics.COSINE.value,\n",
    "    params={\"nlist\": 1024})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4b45f5a-e583-4d77-9640-75842211fefa",
   "metadata": {},
   "source": [
    "#### Define Sparse Vector Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d970a35b-f9b2-4f8f-93ef-8de5c83c31b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "index_params.add_index(\n",
    "    field_name=\"sparse_embedding\",\n",
    "    index_name=\"sparse_inverted_index\",\n",
    "    index_type=\"SPARSE_INVERTED_INDEX\",\n",
    "    metric_type=KeywordSearchMetrics.BM25.value,\n",
    "    params={\"inverted_index_algo\": \"DAAT_MAXSCORE\", \"bm25_k1\": 1.2, \"bm25_b\": 0.75})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "0d45a6ad-2009-4e30-b38d-73266da98a06",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'field_name': 'embedding', 'index_type': 'IVF_FLAT', 'index_name': 'dense_embedding_ivf_flat', 'nlist': 1024, 'metric_type': 'COSINE'},\n",
       " {'field_name': 'sparse_embedding', 'index_type': 'SPARSE_INVERTED_INDEX', 'index_name': 'sparse_inverted_index', 'inverted_index_algo': 'DAAT_MAXSCORE', 'bm25_k1': 1.2, 'bm25_b': 0.75, 'metric_type': 'BM25'}]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index_params"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22a260da-8869-40bb-9cbf-28a73e8cca24",
   "metadata": {},
   "source": [
    "#### Create Collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "51dd4423-240c-4271-bb8c-6270f399a25c",
   "metadata": {},
   "outputs": [],
   "source": [
    "collection_name = \"beam_minilm_256\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "9620b1f2-51fa-491c-ad3f-f0676b9b25f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "client.drop_collection(collection_name=collection_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "e6cf3a1d-265c-44db-aba8-d491fab290d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "client.create_collection(collection_name=collection_name, schema=schema, index_params=index_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "94497411-43d3-4300-98b3-1cb33759738e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.has_collection(collection_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42c1c159-875d-411b-a009-4361301b39f6",
   "metadata": {},
   "source": [
    "## Building the Vector Index: Chunking, Embedding, and Storage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "25c5c202-abe0-4d11-82df-e731f0d6201e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "        if (typeof window.interactive_beam_jquery == 'undefined') {\n",
       "          var jqueryScript = document.createElement('script');\n",
       "          jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
       "          jqueryScript.type = 'text/javascript';\n",
       "          jqueryScript.onload = function() {\n",
       "            var datatableScript = document.createElement('script');\n",
       "            datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
       "            datatableScript.type = 'text/javascript';\n",
       "            datatableScript.onload = function() {\n",
       "              window.interactive_beam_jquery = jQuery.noConflict(true);\n",
       "              window.interactive_beam_jquery(document).ready(function($){\n",
       "                \n",
       "              });\n",
       "            }\n",
       "            document.head.appendChild(datatableScript);\n",
       "          };\n",
       "          document.head.appendChild(jqueryScript);\n",
       "        } else {\n",
       "          window.interactive_beam_jquery(document).ready(function($){\n",
       "            \n",
       "          });\n",
       "        }"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Union[Tuple[apache_beam.pvalue.PCollection[~MLTransformOutputT], apache_beam.pvalue.PCollection[apache_beam.pvalue.Row]], apache_beam.pvalue.PCollection[~MLTransformOutputT]] instead.\n",
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Union[Tuple[apache_beam.pvalue.PCollection[~MLTransformOutputT], apache_beam.pvalue.PCollection[apache_beam.pvalue.Row]], apache_beam.pvalue.PCollection[~MLTransformOutputT]] instead.\n",
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Union[Tuple[apache_beam.pvalue.PCollection[~MLTransformOutputT], apache_beam.pvalue.PCollection[apache_beam.pvalue.Row]], apache_beam.pvalue.PCollection[~MLTransformOutputT]] instead.\n",
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Union[Tuple[apache_beam.pvalue.PCollection[~MLTransformOutputT], apache_beam.pvalue.PCollection[apache_beam.pvalue.Row]], apache_beam.pvalue.PCollection[~MLTransformOutputT]] instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Upserted batch of 5 documents. Result: {'upsert_count': 5, 'primary_keys': ['1_0', '1_1', '2_0', '2_1', '3_0']}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "            <link rel=\"stylesheet\" href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\" integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\" crossorigin=\"anonymous\">\n",
       "            <div id=\"progress_indicator_ef090119901644a31067b90f8d98d385\">\n",
       "              <div class=\"spinner-border text-info\" role=\"status\"></div>\n",
       "              <span class=\"text-info\">Processing... show</span>\n",
       "            </div>\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Upserted batch of 5 documents. Result: {'upsert_count': 5, 'primary_keys': ['1_0', '1_1', '2_0', '2_1', '3_0']}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "            <style>\n",
       "            .p-Widget.jp-OutputPrompt.jp-OutputArea-prompt:empty {\n",
       "              padding: 0;\n",
       "              border: 0;\n",
       "            }\n",
       "            .p-Widget.jp-RenderedJavaScript.jp-mod-trusted.jp-OutputArea-output:empty {\n",
       "              padding: 0;\n",
       "              border: 0;\n",
       "            }\n",
       "            </style>\n",
       "            <link rel=\"stylesheet\" href=\"https://cdn.datatables.net/1.10.20/css/jquery.dataTables.min.css\">\n",
       "            <table id=\"table_df_08499c8cd95657156c076a29cd68a254\" class=\"display\" style=\"display:block\"></table>\n",
       "            <script>\n",
       "              \n",
       "        if (typeof window.interactive_beam_jquery == 'undefined') {\n",
       "          var jqueryScript = document.createElement('script');\n",
       "          jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
       "          jqueryScript.type = 'text/javascript';\n",
       "          jqueryScript.onload = function() {\n",
       "            var datatableScript = document.createElement('script');\n",
       "            datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
       "            datatableScript.type = 'text/javascript';\n",
       "            datatableScript.onload = function() {\n",
       "              window.interactive_beam_jquery = jQuery.noConflict(true);\n",
       "              window.interactive_beam_jquery(document).ready(function($){\n",
       "                \n",
       "            var dt;\n",
       "            if ($.fn.dataTable.isDataTable(\"#table_df_08499c8cd95657156c076a29cd68a254\")) {\n",
       "              dt = $(\"#table_df_08499c8cd95657156c076a29cd68a254\").dataTable();\n",
       "            } else if ($(\"#table_df_08499c8cd95657156c076a29cd68a254_wrapper\").length == 0) {\n",
       "              dt = $(\"#table_df_08499c8cd95657156c076a29cd68a254\").dataTable({\n",
       "                \n",
       "            bAutoWidth: false,\n",
       "            columns: [{'title': ''}, {'title': 'id'}, {'title': 'content'}, {'title': 'title_and_content'}, {'title': 'metadata'}, {'title': 'embedding'}],\n",
       "            destroy: true,\n",
       "            responsive: true,\n",
       "            columnDefs: [\n",
       "              {\n",
       "                targets: \"_all\",\n",
       "                className: \"dt-left\"\n",
       "              },\n",
       "              {\n",
       "                \"targets\": 0,\n",
       "                \"width\": \"10px\",\n",
       "                \"title\": \"\"\n",
       "              }\n",
       "            ]\n",
       "              });\n",
       "            } else {\n",
       "              return;\n",
       "            }\n",
       "            dt.api()\n",
       "              .clear()\n",
       "              .rows.add([{1: '1_0', 2: 'Apache Beam is an open-source framework that provides a consistent programming model for both batch and streaming data processing. Developed originally by Google, it allows developers to write pipelines that can run on multiple engines, such as Apache Flink, Spark, and Google Cloud Dataflow. Beam uses abstractions like PCollections (data containers) and PTransforms (operations) to define the flow of data. The framework promotes portability through its runner architecture, letting the same pipeline execute on different backends. Support for multiple SDKs, including Java and Python, makes it accessible for a broad audience. Key features include support for event time, windowing, triggers, and stateful processing, which are essential for handling real-time data effectively. Beam is ideal for building ETL jobs, real-time analytics, and machine learning data pipelines. It helps teams focus on logic rather than infrastructure, offering flexibility and scalability in handling unbounded and bounded data sources. Apache Beam also supports a wide range of connectors for both input and output, including Kafka, BigQuery, and JDBC-based systems. This makes it easy to integrate Beam into existing data ecosystems. Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing.', 3: 'Apache Beam: Unified Model for Batch and Streaming Data. Apache Beam is an open-source framework that provides a consistent programming model for both batch and streaming data processing. Developed originally by Google, it allows developers to write pipelines that can run on multiple engines, such as Apache Flink, Spark, and Google Cloud Dataflow. Beam uses abstractions like PCollections (data containers) and PTransforms (operations) to define the flow of data. The framework promotes portability through its runner architecture, letting the same pipeline execute on different backends. Support for multiple SDKs, including Java and Python, makes it accessible for a broad audience. Key features include support for event time, windowing, triggers, and stateful processing, which are essential for handling real-time data effectively. Beam is ideal for building ETL jobs, real-time analytics, and machine learning data pipelines. It helps teams focus on logic rather than infrastructure, offering flexibility and scalability in handling unbounded and bounded data sources. Apache Beam also supports a wide range of connectors for both input and output, including Kafka, BigQuery, and JDBC-based systems. This makes it easy to integrate Beam into existing data ecosystems. Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing.', 4: \"{'title': 'Apache Beam: Unified Model for Batch and Streaming Data', 'keywords': ['Apache Beam', 'stream processing', 'batch processing', 'data pipelines', 'SDK'], 'tags': ['Data Engineering', 'Open Source', 'Streaming', 'Batch', 'Big Data']}\", 5: '[-0.06466388702392578, -0.029533877968788147, -0.04261693358421326, 0.01254373136907816, 0.002130179898813367, -0.08420056104660034, -0.035637639462947845, -0.016679927706718445, -0.05082850158214569, -0.08004148304462433, -0.071900874376297, 0.024430863559246063, -0.04954373463988304, -0.048748258501291275, -0.011967036873102188, 0.006303164642304182, 0.05958455801010132, 0.0028353261295706034, -0.07481587678194046, -0.11607144773006439, -0.03259394317865372, -0.041554611176252365, -0.03504106029868126, -0.001439800951629877, -0.02360020950436592, 0.059943076223134995, -0.01519234012812376, -0.034146957099437714, 0.004027337301522493, -0.054018571972846985, 0.02313961647450924, -0.031018856912851334, -0.05701017752289772, 0.040523823350667953, -0.04349478334188461, 0.021394044160842896, 0.007265422493219376, -0.04372619464993477, -0.12264154851436615, -0.09200466424226761, 0.03763662651181221, -0.05324647203087807, -0.09666550904512405, 0.022716403007507324, -0.08052070438861847, -0.07522332668304443, -0.01606123149394989, -0.11872679740190506, -0.008652692660689354, 0.07510612159967422, -0.08550882339477539, -0.06455493718385696, 0.009230186231434345, 0.07337071001529694, -0.007284300867468119, 0.053089335560798645, 0.025012554600834846, 0.00547427823767066, -0.011389803141355515, 0.02432127669453621, -0.024486027657985687, -0.04197389632463455, -0.00921629648655653, 0.05588779225945473, -0.04310512915253639, -0.05197839438915253, 0.06383068859577179, 0.051967501640319824, 0.1240178570151329, -0.10710014402866364, -0.056561537086963654, 0.06153789162635803, -0.01450709905475378, 0.023290887475013733, -0.0773656889796257, 0.0013129215221852064, 0.0134525615721941, -0.002708431100472808, 0.0018889567581936717, 0.007732840720564127, 0.00323132099583745, 0.018673783168196678, -0.09686124324798584, 0.009700699709355831, 0.023337548598647118, -0.0077779777348041534, -0.013852821663022041, 0.11304262280464172, -0.03682168573141098, 0.026285415515303612, -0.0008936264785006642, 0.02425266243517399, 0.07036064565181732, -0.014448655769228935, -0.004841167479753494, 0.03809678927063942, 0.02485431730747223, -0.035138584673404694, -0.0029179861303418875, 0.04480774700641632, -0.030326586216688156, 0.045793332159519196, 0.05787701532244682, -0.0073225838132202625, -0.02137342281639576, -0.08531729876995087, -0.10222272574901581, 0.1223769336938858, -0.02004469744861126, -0.07306617498397827, 0.0631437674164772, -0.008705795742571354, -0.002412058413028717, -0.05419433116912842, 0.038417965173721313, -0.0283599141985178, -0.09585224837064743, -0.0484754703938961, -0.013657975010573864, 0.03131190687417984, 0.031260404735803604, 0.017582444474101067, -0.013743625022470951, 0.013575058430433273, 0.06819900870323181, 0.009789698757231236, -0.08053038269281387, 4.226467530520472e-33, 0.0030900901183485985, -0.015385894104838371, -0.042228877544403076, -0.07484018802642822, 0.05486077442765236, -0.03897050768136978, 0.044820234179496765, 0.031497374176979065, -0.09605666995048523, -0.027088796719908714, 0.03788488730788231, 0.06610210984945297, 0.05553589016199112, -0.004346764646470547, 0.06688157469034195, -0.0606110654771328, -0.042873118072748184, 0.07787125557661057, 0.06925395131111145, 0.01790660433471203, 0.05190691724419594, -0.049133073538541794, -0.028836533427238464, 0.03326796367764473, 0.016870610415935516, 0.03505547344684601, 0.032689593732357025, 0.02542627975344658, -0.025435226038098335, 0.04890316352248192, 0.002155381953343749, -0.020533351227641106, -0.003533762414008379, 0.0411994643509388, 0.055960118770599365, -0.07614350318908691, -0.054680973291397095, -0.09691374748945236, 0.009100464172661304, -0.010418709367513657, -0.05354749411344528, 0.0027101878076791763, -0.03828197717666626, -0.01852724887430668, -0.05890074372291565, -0.044915273785591125, -0.013576041907072067, 0.035576049238443375, -0.04080145061016083, 0.005122833885252476, 0.09771314263343811, -0.003387728938832879, 0.11922183632850647, 0.04530109465122223, 0.05175572261214256, 0.057451214641332626, 0.02983781509101391, 0.013441174291074276, 0.04309932142496109, 0.05667578801512718, -0.0833202451467514, -0.0006599651533178985, -0.053148698061704636, 0.03123246692121029, 0.01695791445672512, -0.02130052074790001, 0.01621287688612938, 0.014019771479070187, 0.005802274215966463, 0.030075233429670334, 0.007117005065083504, 0.04007411375641823, 0.04097031056880951, 0.07147867977619171, 0.03629567474126816, 0.0022272050846368074, 0.022832626476883888, 0.002338982652872801, -0.050566937774419785, 0.03600058704614639, -0.05889451876282692, -0.0052792844362556934, 0.03501333296298981, 0.004061616957187653, 0.036560703068971634, -0.0014219945296645164, -0.057203203439712524, 0.033465780317783356, -0.07556362450122833, 0.01921098865568638, -0.024971656501293182, 0.05516916885972023, 0.07144436985254288, 0.00015671631263103336, -0.01950802654027939, -3.650351137026746e-33, 0.017354466021060944, 0.059810806065797806, -0.05945239216089249, 0.06891019642353058, 0.11440429836511612, 0.009432020597159863, 0.009542316198348999, -0.04723283648490906, 0.005809252616018057, -0.03874228522181511, -0.078343965113163, -0.013072866015136242, 0.008306140080094337, 0.0019391959067434072, 0.0013291530776768923, -0.016417009755969048, -0.00790349580347538, -0.1526392549276352, -0.014709247276186943, 0.0038721419405192137, -0.01859547197818756, 0.05286412686109543, -0.02104310132563114, 0.016759684309363365, -0.012588641606271267, -0.04436327889561653, -0.061210691928863525, -0.05035687983036041, -0.02498915046453476, 0.0033777521457523108, 0.024367185309529305, -0.018248409032821655, 0.025040041655302048, -0.10882879048585892, 0.006693651434034109, -0.005851115100085735, 0.1111128181219101, 0.007033593021333218, 0.03136799484491348, 0.06491357088088989, 0.09991703927516937, 0.013178274035453796, -0.02550503984093666, -0.009967111982405186, -0.054682184010744095, 0.09739664196968079, -0.05937360227108002, 0.11696966737508774, -0.034573908895254135, -0.021528026089072227, -0.10162897408008575, 0.04898342490196228, -0.019914917647838593, 0.0065137529745697975, 0.03217782452702522, -0.06474792957305908, 0.07378212362527847, 0.00545730022713542, -0.10361640155315399, 0.0034713854547590017, 0.01891847513616085, -0.07079721987247467, 0.07830165326595306, 0.02078700065612793, 0.01653195545077324, -0.01404052134603262, 0.021191304549574852, -0.011555657722055912, -0.15100634098052979, -0.02106490358710289, 0.07326526194810867, -0.09215068072080612, 0.013960559852421284, 0.09145322442054749, 0.0014025433920323849, -0.04531499743461609, -0.00909416563808918, -0.03184480965137482, 0.041091516613960266, 0.12100711464881897, 0.024355394765734673, 0.07038372755050659, 0.0430283360183239, 0.03169526159763336, 0.10590188205242157, 0.027500400319695473, -0.0008741550846025348, -0.015190372243523598, 0.0063900393433868885, -0.01886691153049469, -0.05316048115491867, 0.08248165994882584, -0.06078287959098816, 0.06999880820512772, 0.054216716438531876, -4.799430541879701e-08, 0.024673016741871834, 0.02466781623661518, -0.08306656777858734, 0.00029196596005931497, -0.0018005361780524254, 0.005545003805309534, 0.02234015241265297, 0.13406866788864136, 0.04606040194630623, -0.012594856321811676, 0.0772823765873909, -0.07090407609939575, -0.04751046374440193, 0.022145597264170647, 0.07882999628782272, 0.06339012086391449, 0.06228167191147804, -0.040852054953575134, -0.035441990941762924, -0.02966366894543171, 0.04845830798149109, -0.000818374683149159, -0.00830867700278759, 0.05047774314880371, -0.051014143973588943, 0.009566603228449821, 0.12313132733106613, 0.03784331679344177, 0.02365674078464508, -0.06346344202756882, -0.054321423172950745, -0.01641339249908924, 0.0401403084397316, 0.023231539875268936, 0.03016555868089199, 0.03899690881371498, 0.015127046965062618, 0.01934240385890007, 0.02179247885942459, 0.06229304149746895, 0.001949132769368589, 0.05174288898706436, -0.051228415220975876, -0.008371411822736263, -0.022170983254909515, 0.03172885254025459, -0.04765719547867775, -0.031184902414679527, -0.029825732111930847, 0.052426744252443314, 0.011883549392223358, -0.04567771404981613, 0.005567905493080616, 0.022183997556567192, 0.10563989728689194, 0.06372497975826263, 0.04733740910887718, -0.08152730017900467, 0.08148864656686783, 0.059733033180236816, 0.036120910197496414, 0.034422941505908966, 0.03181681036949158, 0.0001891597785288468]', 0: 0}, {1: '1_1', 2: \"Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing. The concept of runners enables developers to write once and run anywhere, which is particularly appealing for organizations that want to avoid vendor lock-in. The Beam model is based on a unified programming model that decouples pipeline logic from execution. This makes it easier to reason about time and state in both batch and streaming pipelines. Advanced features like late data handling, watermarks, and session windowing allow for more accurate and meaningful processing of real-world data. Beam also integrates with orchestration tools and monitoring systems, allowing for production-grade deployments. Community support and contributions have grown significantly, making Beam a stable and evolving ecosystem. Many cloud providers offer native support for Beam pipelines, and it's increasingly a core component in modern data platform architectures.\", 3: \"Apache Beam: Unified Model for Batch and Streaming Data. Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing. The concept of runners enables developers to write once and run anywhere, which is particularly appealing for organizations that want to avoid vendor lock-in. The Beam model is based on a unified programming model that decouples pipeline logic from execution. This makes it easier to reason about time and state in both batch and streaming pipelines. Advanced features like late data handling, watermarks, and session windowing allow for more accurate and meaningful processing of real-world data. Beam also integrates with orchestration tools and monitoring systems, allowing for production-grade deployments. Community support and contributions have grown significantly, making Beam a stable and evolving ecosystem. Many cloud providers offer native support for Beam pipelines, and it's increasingly a core component in modern data platform architectures.\", 4: \"{'title': 'Apache Beam: Unified Model for Batch and Streaming Data', 'keywords': ['Apache Beam', 'stream processing', 'batch processing', 'data pipelines', 'SDK'], 'tags': ['Data Engineering', 'Open Source', 'Streaming', 'Batch', 'Big Data']}\", 5: '[-0.018973594531416893, -0.038047756999731064, 0.009853314608335495, 0.017600156366825104, 0.01497685257345438, -0.06582952290773392, -0.07951309531927109, -0.02883726730942726, -0.041912175714969635, -0.04387732222676277, -0.08884589374065399, 0.053467925637960434, -0.05935873091220856, -0.03815134987235069, 0.01334142591804266, -0.0048108333721756935, 0.0428265780210495, -0.037232715636491776, -0.11089688539505005, -0.08068843185901642, -0.05602698773145676, -0.06249658390879631, -0.09883567690849304, 0.05031457543373108, -0.020239757373929024, 0.07551726698875427, -0.06086629256606102, -0.05752619728446007, 0.01581764407455921, -0.04817376285791397, -0.0023335570003837347, -0.04845050722360611, -0.0006991951959207654, 0.06703019887208939, -0.0055598048493266106, 0.016426336020231247, 0.02817388065159321, -0.07312498986721039, -0.07833398133516312, -0.06406418979167938, 0.028878629207611084, -0.0944715365767479, -0.09821832925081253, 0.044137436896562576, -0.009284254163503647, -0.07901174575090408, -0.04781392216682434, -0.09630963951349258, -0.07280350476503372, 0.08982539176940918, -0.04611873999238014, -0.08185379952192307, 0.02143690548837185, 0.035766519606113434, -0.029929867014288902, 0.0987226590514183, 0.053078461438417435, 0.06040719151496887, 0.0037211780436336994, 0.01677882857620716, -0.0024818717502057552, -0.06774415075778961, -0.022503690794110298, 0.047713641077280045, 0.03527110442519188, -0.033364977687597275, 0.009873880073428154, 0.04558619111776352, 0.11955580115318298, -0.09005599468946457, -0.010989201255142689, 0.031770385801792145, 0.008936704136431217, 0.00538625055924058, -0.06366196274757385, 0.03144121542572975, 0.01911686360836029, -0.04297826811671257, -0.012642892077565193, -0.007212298922240734, 0.03595840930938721, -0.018883714452385902, -0.06322586536407471, 0.025581810623407364, 0.02180952951312065, 0.06087557598948479, 0.04372349753975868, 0.057813845574855804, 0.01641206257045269, -0.026707131415605545, -0.050563156604766846, 0.004152100067585707, 0.09263436496257782, -0.015737539157271385, -0.0007108922582119703, 0.02848093770444393, -0.00806748028844595, -0.05122291296720505, 0.005889056716114283, 0.04067201167345047, -0.00850051362067461, 0.02599603682756424, 0.10257657617330551, 0.01619791053235531, 0.00833812728524208, -0.08265282213687897, -0.04061337560415268, 0.1259007751941681, -0.027060644701123238, -0.06887192279100418, 0.04976821318268776, -0.015719763934612274, 0.023927684873342514, -0.036306049674749374, 0.05325644835829735, -0.06689215451478958, -0.09015952050685883, -0.008463169448077679, -0.03694949671626091, 0.06418510526418686, 0.04649828001856804, -0.004484950564801693, -0.011765721254050732, -0.03851206228137016, 0.06064915657043457, 0.029278069734573364, -0.05129281431436539, 3.0011401170759893e-33, 0.005440344102680683, -0.04820927232503891, -0.03920764848589897, -0.043692316859960556, 0.11088904738426208, -0.04808446019887924, 0.049140360206365585, 0.011111056432127953, -0.10306145995855331, -0.02553553320467472, 0.05190734192728996, 0.05066768079996109, 0.04280221089720726, 0.03775731474161148, 0.04095909371972084, -0.08128233999013901, -0.022233635187149048, 0.073038749396801, 0.0646248310804367, 0.04169369861483574, 0.06933081150054932, -0.05534159019589424, -0.03268785774707794, -0.005426599644124508, 0.06439580768346786, -0.01288049016147852, 0.04870288819074631, 0.026471689343452454, -0.005027782171964645, 0.02150837890803814, 0.005291462875902653, 0.0027808905579149723, 0.03742994740605354, 0.01261330209672451, 0.06765583157539368, -0.03537099435925484, -0.017706256359815598, -0.09280066192150116, 0.0316665880382061, 0.0168000478297472, -0.029039543122053146, 0.015987912192940712, -0.07633719593286514, -0.024627406150102615, -0.040600694715976715, -0.027892230078577995, -0.033759672194719315, -0.002126255538314581, -0.011880690231919289, -0.028114158660173416, 0.08075553923845291, 0.06590402126312256, 0.04470798373222351, 0.04803197458386421, 0.07680407166481018, -0.03015856444835663, 0.05029886215925217, -0.032047562301158905, 0.07621696591377258, 0.03183460608124733, -0.031034624204039574, -0.024675123393535614, -0.07854422181844711, 0.01856403425335884, -0.007004621904343367, -0.03843999654054642, 0.047416381537914276, 0.0338081531226635, 0.026617716997861862, 0.03181052953004837, -0.03686947375535965, -0.01137223094701767, -0.02098255045711994, 0.0560978427529335, 0.032936133444309235, -0.017804542556405067, -0.012492518872022629, 0.020227478817105293, -0.02718948945403099, 0.015233292244374752, -0.03476536273956299, -0.03465321287512779, -0.009879112243652344, 0.011551113799214363, 0.04184458777308464, -0.0032065215054899454, -0.0724199116230011, -0.002717012306675315, -0.11437557637691498, -0.0005933393258601427, -0.009862713515758514, 0.042416442185640335, 0.04581250995397568, 0.01821465790271759, 0.017902765423059464, -2.764706693481364e-33, 0.0022231105249375105, 0.014193677343428135, -0.07299257069826126, 0.08803818374872208, 0.08409763872623444, -0.03365476429462433, -0.0013658803654834628, -0.0987565815448761, -0.0033523032907396555, -0.02895462140440941, -0.09524943679571152, -0.0017201234586536884, -0.03854745253920555, 0.03291812166571617, -0.03718029335141182, -0.018591750413179398, -0.005976638291031122, -0.12552377581596375, -0.006126183085143566, -0.002084973966702819, 0.0248258113861084, 0.03312137722969055, -0.04124243184924126, 0.009689252823591232, -0.011249818839132786, -0.0053729587234556675, -0.0344584584236145, -0.058878734707832336, 0.017762307077646255, -0.052064865827560425, 0.0032393925357609987, 0.002994223264977336, 0.020968977361917496, -0.1437208503484726, -0.0004785321361850947, 0.030647477135062218, 0.0286928191781044, 0.044096317142248154, 0.019514000043272972, -0.0012711107265204191, 0.06709816306829453, 0.01644347794353962, 0.02145499736070633, -0.03152571991086006, -0.04268411546945572, 0.1273767203092575, 0.02085105888545513, 0.051103588193655014, -0.09107273072004318, -0.022750038653612137, -0.03109086863696575, -0.01736641116440296, 0.0010191221954301, 0.0037347853649407625, 0.03477935120463371, -0.08281423896551132, 0.05154472216963768, -0.026259252801537514, -0.11400901526212692, 0.025394976139068604, 0.04734973981976509, -0.03389143571257591, 0.12601801753044128, -0.00430592754855752, -0.024579815566539764, -0.0002861841639969498, 0.017392922192811966, -0.013216436840593815, -0.11752787232398987, 0.012718579731881618, 0.013284187763929367, -0.08756081759929657, -0.03979286178946495, 0.09630093723535538, -0.033030349761247635, -0.06131284683942795, -0.03974708914756775, -0.05123161897063255, -0.004189329221844673, 0.09414182603359222, -0.06169441342353821, 0.05233728513121605, 0.032566361129283905, 0.014203887432813644, 0.09891701489686966, 0.04159114882349968, 0.04394884407520294, 0.002195159438997507, -0.0052054510451853275, 0.034669600427150726, -0.057895563542842865, 0.09207107126712799, -0.07015221565961838, 0.05211507901549339, -0.02817714586853981, -4.9893632336761584e-08, -0.00896522868424654, 0.04456833750009537, -0.0719098225235939, 0.04353230446577072, 0.0015314699849113822, -0.010044410824775696, 0.06597968190908432, 0.09209451824426651, 0.09022504091262817, -0.010507077910006046, 0.07367205619812012, -0.09957010298967361, -0.028277182951569557, 0.0457858182489872, 0.12363816797733307, 0.04674151912331581, 0.04391893744468689, 0.0015561962500214577, -0.0487351231276989, -0.04268272593617439, 0.006729984190315008, 0.04273779317736626, 0.0370924174785614, 0.07646933197975159, -0.016536453738808632, 0.030621210113167763, 0.09810485690832138, 0.06789854168891907, 0.019589059054851532, -0.003846997395157814, -0.016470609232783318, 0.0031430134549736977, 0.06656448543071747, 0.009443582966923714, 0.06827470660209656, 0.04586328566074371, 0.03099874034523964, 0.01014702022075653, 0.01989901438355446, 0.030771005898714066, -0.0008424646803177893, 0.03357389569282532, 0.012780584394931793, 0.019024720415472984, -0.003248361637815833, 0.0033370780292898417, -0.059965137392282486, -0.004392987582832575, -0.04737864434719086, -0.006688870955258608, 0.01745840534567833, -0.06501461565494537, -0.0549679696559906, 0.07742945104837418, 0.1438921093940735, 0.05695037171244621, 0.06452634185552597, -0.06825384497642517, 0.0938495546579361, 0.08684926480054855, 0.00600523641332984, -0.02143688313663006, 0.046269144862890244, -0.03446299210190773]', 0: 1}, {1: '2_0', 2: 'Google Cloud Dataflow is a fully managed service that runs Apache Beam pipelines in the cloud. It abstracts away infrastructure management and handles dynamic scaling, load balancing, and fault tolerance. Developers can focus on writing data logic using the Beam SDK and deploy it easily to Google Cloud. Dataflow supports both batch and stream processing and integrates seamlessly with other Google services like BigQuery, Pub/Sub, and Cloud Storage. Its autoscaling capabilities allow it to adapt to changing data volumes, optimizing for cost and performance. Features like monitoring dashboards, job templates, and built-in logging make it suitable for both development and production use. With support for event time processing, stateful functions, and windowing, Dataflow is well-suited for real-time analytics and data transformation tasks. It’s a key component for architects building scalable, cloud-native data platforms. Dataflow also offers templates for common ETL tasks, helping teams get started quickly with minimal setup. Its integration with Cloud Functions and Cloud Composer enables event-driven and orchestrated workflows. Security and compliance are built-in with IAM roles, encryption at rest and in transit, and audit logging, making it suitable for enterprise environments.', 3: 'Google Cloud Dataflow: Run Apache Beam in the Cloud. Google Cloud Dataflow is a fully managed service that runs Apache Beam pipelines in the cloud. It abstracts away infrastructure management and handles dynamic scaling, load balancing, and fault tolerance. Developers can focus on writing data logic using the Beam SDK and deploy it easily to Google Cloud. Dataflow supports both batch and stream processing and integrates seamlessly with other Google services like BigQuery, Pub/Sub, and Cloud Storage. Its autoscaling capabilities allow it to adapt to changing data volumes, optimizing for cost and performance. Features like monitoring dashboards, job templates, and built-in logging make it suitable for both development and production use. With support for event time processing, stateful functions, and windowing, Dataflow is well-suited for real-time analytics and data transformation tasks. It’s a key component for architects building scalable, cloud-native data platforms. Dataflow also offers templates for common ETL tasks, helping teams get started quickly with minimal setup. Its integration with Cloud Functions and Cloud Composer enables event-driven and orchestrated workflows. Security and compliance are built-in with IAM roles, encryption at rest and in transit, and audit logging, making it suitable for enterprise environments.', 4: \"{'title': 'Google Cloud Dataflow: Run Apache Beam in the Cloud', 'keywords': ['Google Cloud', 'Dataflow', 'Apache Beam', 'serverless', 'stream and batch'], 'tags': ['Cloud Computing', 'Data Pipelines', 'Google Cloud', 'Serverless', 'Enterprise']}\", 5: '[-0.06922302395105362, 0.023161506280303, 0.045865464955568314, -0.043821707367897034, 0.012096977792680264, -0.05220700427889824, 0.001474533462896943, -0.060477327555418015, 0.02311597764492035, 0.006155427545309067, -0.09254368394613266, -0.026979118585586548, 0.007794504053890705, -0.03784167021512985, -0.008246206678450108, -0.014848004095256329, 0.06783697754144669, -0.033511579036712646, -0.11154574900865555, -0.08980037271976471, -0.06648188084363937, -0.020473342388868332, -0.07527496665716171, 0.03519774600863457, -0.038868505507707596, 0.0657028779387474, -0.014987119473516941, -0.08130184561014175, 0.009730864316225052, -0.05540500208735466, -0.013050466775894165, -0.04525986686348915, -0.004191172309219837, 0.13747315108776093, -0.04854223132133484, -0.010875235311686993, 0.0329212062060833, -0.011579407379031181, -0.07976572215557098, -0.07062806934118271, 0.04519709199666977, -0.1190754696726799, -0.0873320996761322, -0.05216065049171448, -0.051656756550073624, -0.03193237632513046, -0.029147827997803688, -0.12397291511297226, -0.049637433141469955, 0.0897335410118103, -0.02015087939798832, -0.0870516300201416, 0.0478004589676857, 0.061090752482414246, -0.07045301049947739, 0.06690108776092529, 0.05036143586039543, -0.006564647890627384, 0.028151128441095352, 0.0011042800033465028, -0.03730063512921333, -0.04508212208747864, -0.024164803326129913, 0.03232676535844803, -0.03507281094789505, 0.0007927772821858525, 0.06290047615766525, 0.013078553602099419, 0.0635475292801857, -0.0221528559923172, 0.0028895079158246517, 0.02719644084572792, -0.03433110564947128, -0.015487424097955227, -0.07674422860145569, -0.04146977514028549, -0.044433802366256714, -0.04066569358110428, -0.0005891985492780805, -0.02918723039329052, 0.04124194011092186, 0.06517095863819122, -0.030084604397416115, 0.08247551321983337, -0.021744975820183754, -0.06545085459947586, -0.015583222731947899, 0.07846656441688538, 0.017796527594327927, -0.0050199697725474834, -0.012236016802489758, 0.040639594197273254, 0.04380558431148529, -0.03247296065092087, 0.07249102741479874, 0.023049896582961082, -0.04953192174434662, -0.05010318011045456, 0.05910531431436539, -0.01007019355893135, -0.012454433366656303, 0.09351649135351181, 0.05062120407819748, -0.017186110839247704, 0.03257777541875839, -0.036310818046331406, -0.07265055179595947, 0.085996612906456, 0.0035129461903125048, -0.02146696299314499, 0.04926305636763573, 0.06111319735646248, 0.016581900417804718, -0.027423610910773277, 0.06676501780748367, 0.019711701199412346, -0.07355605065822601, -0.05667809024453163, -0.07576645165681839, 0.10831525176763535, 0.0639774426817894, 0.019096162170171738, -0.045362140983343124, 0.023760955780744553, 0.052066754549741745, 0.0030979060102254152, -0.03407223895192146, 2.9031551324242667e-33, 0.030751287937164307, -0.01548618171364069, -0.0002451580367051065, -0.06648711115121841, 0.10106846690177917, 0.022745406255126, -0.0013853602577000856, -0.010138805955648422, -0.061261966824531555, -0.016061361879110336, -0.02694455347955227, 0.10520829260349274, -0.013813639059662819, 0.006083263549953699, 0.05271698161959648, -0.01239310298115015, -0.031855691224336624, 0.09228593111038208, 0.09149166941642761, 0.02157820574939251, 0.025037162005901337, -0.09339465945959091, 0.0025386337656527758, 0.03986256942152977, 0.06561947613954544, 0.003864482045173645, 0.07566480338573456, -0.005065985955297947, 0.05211842060089111, 0.01600526086986065, -0.03600906580686569, -0.04902055859565735, 0.01332682091742754, 0.047950517386198044, -0.02850937284529209, -0.04355623200535774, -0.10912280529737473, -0.04597802832722664, 0.04209766909480095, 0.037293560802936554, -0.003396362764760852, 0.013617961667478085, -0.07035071402788162, -0.05462810769677162, -0.0700947716832161, 0.015253187157213688, -0.008539300411939621, -0.009663798846304417, 0.04764934629201889, -0.006874932907521725, 0.12136242538690567, 0.017473505809903145, 0.03491348400712013, -0.006369094364345074, 0.06318545341491699, 0.030066514387726784, 0.06774647533893585, -0.023543784394860268, 0.001241658697836101, 0.05070498585700989, -0.06009829044342041, -0.01978699490427971, -0.03339686989784241, 0.05812661722302437, 0.00752455135807395, -0.083124078810215, 0.014432292431592941, 0.09036456793546677, 0.014141008257865906, -0.013721432536840439, -0.028522038832306862, -0.006297665182501078, 0.08145670592784882, 0.07038164883852005, 0.0544198676943779, -0.008289210498332977, -0.007178945001214743, 0.015230788849294186, -0.07144331187009811, 0.04001161456108093, -0.047253143042325974, -0.017992176115512848, 0.053590234369039536, 0.0423310287296772, 0.0031884820200502872, -0.03413547948002815, -0.014060970395803452, 0.04878852143883705, -0.12464924901723862, 0.019769102334976196, -0.05688019469380379, -0.019350357353687286, 0.10566561669111252, 0.0054002427496016026, -0.05828821286559105, -3.197660181001421e-33, -0.015062433667480946, 0.005063250195235014, -0.03854287415742874, 0.09942492842674255, 0.1009494811296463, 0.007028763648122549, 0.0347612090408802, 0.001351911574602127, 0.01721402257680893, 0.02427109330892563, -0.05357150733470917, 0.007844974286854267, 0.004905079957097769, -0.00778529979288578, -0.007310265675187111, -0.03678283840417862, -0.009034755639731884, -0.11026682704687119, -0.052736155688762665, -0.02109014242887497, -0.018955320119857788, 0.01524350605905056, -0.017219502478837967, -0.01637597195804119, -0.024668259546160698, 0.005761103704571724, -0.061775751411914825, -0.02436325140297413, 0.019888758659362793, 0.012886084616184235, -0.031120268628001213, 0.034060172736644745, 0.058737706393003464, -0.07428651303052902, 0.06617841124534607, -0.07656872272491455, 0.06839325278997421, 0.03815082460641861, 0.017387909814715385, -0.008589591830968857, -0.012417268939316273, -0.007639465853571892, 0.03485056012868881, -0.0672459825873375, -0.036845918744802475, 0.10244598239660263, 0.00019573078316170722, 0.06977578997612, -0.0555693581700325, -0.05785490944981575, -0.0764404758810997, 0.032553814351558685, -0.03484220430254936, 0.033937014639377594, 0.009329676628112793, -0.0365307480096817, 0.020915787667036057, -0.021535539999604225, -0.11950027197599411, 0.0016304069431498647, -0.01581830158829689, -0.013064299710094929, 0.12702429294586182, 0.01419518981128931, -0.023599980399012566, 0.02876134403049946, 0.0182856023311615, -0.002988226478919387, -0.13134202361106873, 0.03873820602893829, 0.05212586373090744, -0.08891689777374268, -0.0006976487347856164, 0.05933821201324463, 0.002799754962325096, -0.023531176149845123, 0.031348053365945816, -0.13147170841693878, 0.03059895522892475, 0.030061455443501472, 0.01293276622891426, 0.05505232885479927, -0.010720016434788704, 0.021187424659729004, 0.06960820406675339, -0.002950818045064807, 0.005676741246134043, -0.027272380888462067, 0.04858771711587906, 0.0734899491071701, -0.1088961735367775, 0.00760533194988966, -0.11584514379501343, 0.029936376959085464, 0.01612277142703533, -4.575245071691825e-08, -0.01872018165886402, 0.047710951417684555, -0.04508286714553833, 0.000703521363902837, 0.007688414771109819, -0.031201617792248726, 0.05539138987660408, 0.11052795499563217, 0.04336031526327133, -0.009162266738712788, 0.04374673217535019, -0.11823780834674835, -0.01510899793356657, -0.018336022272706032, 0.05745872110128403, 0.002433251589536667, 0.07766743004322052, -0.04140744358301163, -0.02027367614209652, -0.05821099132299423, -0.00770132802426815, 0.030263029038906097, -0.01441414374858141, 0.024740438908338547, -0.02200401946902275, 0.00046951076365076005, 0.09944584965705872, 0.013641657307744026, 0.013016611337661743, -0.06615390628576279, -0.040024783462285995, 0.0012006880715489388, 0.035945646464824677, 0.0238544549793005, 0.04967077821493149, -0.03130926564335823, 0.005723670590668917, -0.02381395734846592, 0.03850293904542923, 0.0029672004748135805, 0.04136791080236435, 0.05762109532952309, -0.015518826432526112, -0.025523079559206963, -0.012718209996819496, -0.016003094613552094, -0.03060447797179222, 0.0014687910443171859, -0.00880576390773058, 0.115822434425354, 0.01786453276872635, -0.045547958463430405, 0.02170303650200367, 0.09815438836812973, 0.10668042302131653, 0.034177426248788834, 0.03066622093319893, -0.10539556294679642, 0.12173125892877579, 0.0512351468205452, 0.03973713889718056, 0.030443686991930008, 0.007508042734116316, -0.03698401898145676]', 0: 2}, {1: '2_1', 2: 'For developers, Dataflow provides local testing capabilities and a unified logging system through Cloud Logging. It also supports SQL-based pipeline definitions using BigQuery, which lowers the barrier to entry for analysts and data engineers. Dataflow’s streaming engine significantly improves performance and reduces costs by decoupling compute and state management. In summary, Google Cloud Dataflow not only simplifies the deployment of Apache Beam pipelines but also enhances them with cloud-native features. Its managed runtime, high availability, and integration with the broader Google Cloud ecosystem make it a powerful tool for modern data processing.', 3: 'Google Cloud Dataflow: Run Apache Beam in the Cloud. For developers, Dataflow provides local testing capabilities and a unified logging system through Cloud Logging. It also supports SQL-based pipeline definitions using BigQuery, which lowers the barrier to entry for analysts and data engineers. Dataflow’s streaming engine significantly improves performance and reduces costs by decoupling compute and state management. In summary, Google Cloud Dataflow not only simplifies the deployment of Apache Beam pipelines but also enhances them with cloud-native features. Its managed runtime, high availability, and integration with the broader Google Cloud ecosystem make it a powerful tool for modern data processing.', 4: \"{'title': 'Google Cloud Dataflow: Run Apache Beam in the Cloud', 'keywords': ['Google Cloud', 'Dataflow', 'Apache Beam', 'serverless', 'stream and batch'], 'tags': ['Cloud Computing', 'Data Pipelines', 'Google Cloud', 'Serverless', 'Enterprise']}\", 5: '[-0.03661690652370453, -0.03150395676493645, 0.06368837505578995, -0.03153130039572716, -0.004471856635063887, -0.09541423618793488, -0.054404497146606445, -0.09089410305023193, -0.050178591161966324, -0.006378731224685907, -0.06700551509857178, -0.0004740693257190287, -0.008150320500135422, -0.07737687230110168, -0.027979711070656776, 0.0010401720646768808, 0.046904824674129486, -0.0015425728633999825, -0.08718173205852509, -0.08254003524780273, -0.04283147677779198, 0.00358956097625196, -0.07748213410377502, 0.04272042214870453, 0.035424426198005676, 0.06755490601062775, -0.05363599956035614, -0.058646537363529205, 0.033957578241825104, -0.02617446333169937, -0.020357120782136917, -0.007250271271914244, -0.059132955968379974, 0.13508857786655426, -0.021537745371460915, -0.026470733806490898, 0.0009676427580416203, 0.008185886777937412, -0.07078035175800323, -0.04654406011104584, 0.07878093421459198, -0.1012917160987854, -0.04706841707229614, -0.035616252571344376, -0.0722387358546257, -0.022064778953790665, -0.00020241140737198293, -0.10148043185472488, -0.03747682273387909, 0.06451018899679184, -0.053426895290613174, -0.07141502946615219, -0.005222890060395002, 0.09770642220973969, -0.04477661848068237, 0.03803936019539833, 0.03609953820705414, 0.011689776554703712, 0.010280258022248745, 0.05735364183783531, -0.048317525535821915, -0.02284780703485012, 0.012211902067065239, 0.0429220050573349, -0.021692665293812752, -0.020572056993842125, 0.07219453901052475, 0.031036250293254852, 0.0844205915927887, -0.06730155646800995, -0.009439797140657902, 0.08722429722547531, -0.05560028925538063, -0.00539100356400013, -0.0876370370388031, -0.009375905618071556, -0.012034732848405838, 0.0021236685570329428, 0.020318949595093727, 0.024571334943175316, 0.018642617389559746, 0.0068883285857737064, -0.032303888350725174, 0.07302466779947281, -0.035076647996902466, -0.03563516214489937, 0.02606920525431633, 0.05076679587364197, 0.02319413423538208, -0.009693698957562447, -0.047582466155290604, 0.002919740043580532, 0.05556236580014229, -0.013434740714728832, -0.015244543552398682, 0.02350369654595852, -0.002612635027617216, -0.03747796639800072, 0.029851974919438362, -0.017929568886756897, 0.005668936297297478, 0.05689527839422226, 0.08259685337543488, -0.04409698396921158, -0.020461028441786766, -0.05634477734565735, -0.06020928546786308, 0.0912451222538948, 0.04807163402438164, -0.0634351521730423, 0.06615909934043884, 0.0577421560883522, 0.02740497700870037, -0.031116239726543427, 0.04045913368463516, 0.018953057006001472, -0.10272610187530518, -0.08337682485580444, -0.09989166259765625, 0.043805431574583054, 0.02688218280673027, 0.014868419617414474, -0.034112751483917236, -0.027920272201299667, 0.029938897117972374, 0.011456538923084736, -0.06435238569974899, 1.9176078117253987e-33, 0.048523493111133575, -0.016604095697402954, 0.02822207100689411, -0.11147268861532211, 0.08810616284608841, 0.003103473922237754, 0.016116252169013023, 0.01928049698472023, -0.09448369592428207, -0.013867964968085289, -0.027936916798353195, 0.07054978609085083, 0.00670698331668973, 0.01044241338968277, 0.03523870185017586, -0.04192784056067467, -0.018141023814678192, 0.07065747678279877, 0.08403744548559189, 0.061018604785203934, 0.04657676815986633, -0.05936289206147194, -0.009776483289897442, 0.007256131153553724, 0.021935366094112396, 0.012536431662738323, 0.056103043258190155, 0.009438068605959415, 0.04698248580098152, 0.0333077572286129, -0.023015402257442474, -0.08556229621171951, -0.007610197179019451, 0.02038782462477684, 0.03791613131761551, -0.07189298421144485, -0.06980198621749878, -0.0571599155664444, 0.007414001040160656, 0.023433992639183998, -0.00479891849681735, 0.01776411198079586, -0.10016501694917679, -0.04505573958158493, -0.039798036217689514, -0.0009126919903792441, -0.029788795858621597, -0.050578705966472626, 0.03168235346674919, -0.01924174092710018, 0.11106050759553909, 0.017998214811086655, 0.03220384940505028, -0.005235073156654835, 0.06567565351724625, 0.03682543709874153, 0.07076107710599899, -0.022777730599045753, 0.0386214479804039, 0.06058921292424202, -0.07981505990028381, -0.01912962831556797, -0.06236041709780693, 0.035301510244607925, 0.005597411189228296, -0.054670218378305435, 0.047188155353069305, 0.08034088462591171, 0.02310079149901867, -0.004662695806473494, -0.009987777099013329, -0.012309315614402294, 0.06983225792646408, 0.06838769465684891, 0.06794392317533493, -0.011658687144517899, 0.00048192118993029, 0.024348696693778038, -0.027921482920646667, -0.008704553358256817, -0.043475326150655746, -0.02925853431224823, 0.0014699678868055344, 0.047023870050907135, 0.008884437382221222, -0.013704565353691578, -0.02721046283841133, 0.018465938046574593, -0.11433548480272293, 0.03573836013674736, -0.08787364512681961, 0.01614609733223915, 0.07766127586364746, -0.027548586949706078, -0.027749354019761086, -2.5456522191796767e-33, -0.051880232989788055, 0.010609125718474388, -0.034008026123046875, 0.1296122968196869, 0.08679812401533127, -0.05832454934716225, 0.07041805237531662, -0.0008646771311759949, 0.007786416448652744, 0.014124558307230473, -0.05393674597144127, 0.0025924458168447018, 0.0018217518227174878, -0.015095573849976063, 0.035310450941324234, -0.028661319985985756, -0.002545525785535574, -0.13136237859725952, -0.023445023223757744, 0.010562043637037277, -0.004121020436286926, 0.019269153475761414, -0.007250993978232145, 0.004447334911674261, -0.03605847805738449, -0.010661615990102291, -0.05064380168914795, -0.002300841500982642, 0.017919018864631653, -0.01914912648499012, 0.02137143909931183, 0.02259998768568039, 0.006458733230829239, -0.11112105846405029, 0.017941024154424667, -0.04479946568608284, 0.09241434186697006, 0.03140066936612129, 0.03923803195357323, -0.009351697750389576, -0.017988082021474838, -0.009879431687295437, 0.040496826171875, -0.03998282924294472, -0.026738766580820084, 0.09971068054437637, -0.008086304180324078, 0.061276037245988846, -0.032826852053403854, -0.0253130029886961, -0.056416332721710205, 0.015237093903124332, -0.02254786528646946, 0.033029746264219284, 0.025828614830970764, -0.038142841309309006, 0.017419449985027313, 0.002233796985819936, -0.138339564204216, -0.03583831340074539, -0.024587463587522507, -0.03458356112241745, 0.10121329873800278, 0.0017955348594114184, -0.02243809774518013, 0.02294073812663555, 0.01986278034746647, 0.013287126086652279, -0.14369899034500122, -0.008195938542485237, -0.0010208070743829012, -0.1172943189740181, -0.004770664963871241, 0.09437159448862076, 0.008753892965614796, -0.0027984054759144783, -0.015055130235850811, -0.050968363881111145, 0.06796887516975403, 0.08026447147130966, 0.0015055668773129582, 0.08716031163930893, 0.003129352815449238, -0.008337263949215412, 0.14612466096878052, -0.011662853881716728, -0.01087844930589199, -0.015590676106512547, 0.04087084159255028, 0.031123900786042213, -0.08632262796163559, 0.04776468500494957, -0.09908502548933029, -0.0096565131098032, 0.0012567397207021713, -3.492227307333451e-08, -0.03131479024887085, 0.039830658584833145, -0.03799265995621681, -0.0024163657799363136, 0.04210447892546654, 0.013701516203582287, 0.053717710077762604, 0.1696431189775467, 0.041706692427396774, -0.03259803727269173, 0.06020280718803406, -0.08854134380817413, -0.013443593867123127, 0.017301753163337708, 0.06332704424858093, 0.007233734242618084, 0.052556343376636505, -0.0778719037771225, -0.02314665913581848, -0.02860150672495365, -0.007205752190202475, 0.03254684805870056, -0.0012815148802474141, 0.02611367031931877, -0.06734127551317215, -0.011466773226857185, 0.16997094452381134, 0.037703197449445724, -0.02536364272236824, -0.06489993631839752, -0.026916049420833588, 0.008996563032269478, 0.007384839002043009, -0.025420036166906357, 0.07738093286752701, -0.005997790489345789, 0.006048841867595911, 0.03464819863438606, 0.05387331545352936, 0.00037185035762377083, 0.04431060329079628, 0.022498739883303642, 0.015599384903907776, -0.03256010636687279, -0.0007507397094741464, -0.021523743867874146, -0.018510466441512108, 0.0167588759213686, -0.005927710328251123, 0.08524975925683975, 0.025901226326823235, -0.05621309205889702, -0.006224010139703751, 0.09505369514226913, 0.1378629207611084, 0.006168277468532324, 0.02945222333073616, -0.1219346821308136, 0.06274287402629852, 0.03979019075632095, 0.043842218816280365, 0.008064117282629013, 0.04107089713215828, -0.007163587491959333]', 0: 3}, {1: '3_0', 2: 'Google Beam is an innovative video communication platform that builds on the research of Project Starline. It uses AI, 3D imaging, and light field rendering to create immersive, lifelike video calls. Designed to replicate in-person interaction, Beam allows users to see life-sized, three-dimensional representations of each other without the need for headsets. This breakthrough makes remote conversations feel natural—capturing facial expressions, eye contact, and subtle gestures that traditional video conferencing often misses. Beam reduces meeting fatigue and enhances engagement, making it ideal for enterprise collaboration, interviews, and virtual presence scenarios. Powered by Google AI, Beam represents a significant leap in communication technology. Major companies like Salesforce, Deloitte, and NEC are already exploring its impact on digital collaboration. Google is partnering with HP to build and distribute Beam hardware, designed to work with existing productivity and video tools. Currently in limited early access for enterprise partners, Google Beam aims to redefine virtual meetings by bridging the gap between digital and physical presence. It’s a promising step toward more human and effective remote interactions.', 3: 'Google Beam: 3D Communication Powered by AI. Google Beam is an innovative video communication platform that builds on the research of Project Starline. It uses AI, 3D imaging, and light field rendering to create immersive, lifelike video calls. Designed to replicate in-person interaction, Beam allows users to see life-sized, three-dimensional representations of each other without the need for headsets. This breakthrough makes remote conversations feel natural—capturing facial expressions, eye contact, and subtle gestures that traditional video conferencing often misses. Beam reduces meeting fatigue and enhances engagement, making it ideal for enterprise collaboration, interviews, and virtual presence scenarios. Powered by Google AI, Beam represents a significant leap in communication technology. Major companies like Salesforce, Deloitte, and NEC are already exploring its impact on digital collaboration. Google is partnering with HP to build and distribute Beam hardware, designed to work with existing productivity and video tools. Currently in limited early access for enterprise partners, Google Beam aims to redefine virtual meetings by bridging the gap between digital and physical presence. It’s a promising step toward more human and effective remote interactions.', 4: \"{'title': 'Google Beam: 3D Communication Powered by AI', 'keywords': ['Google Beam', 'Project Starline', '3D video', 'AI communication', 'real-time meetings'], 'tags': ['AI', 'Communication', '3D Technology', 'Remote Work', 'Enterprise Tech']}\", 5: '[-0.08768321573734283, -0.10005483776330948, 0.03568415343761444, -0.08212734013795853, -0.01738724298775196, -0.0572899766266346, 0.0060171508230268955, -0.06130814179778099, 0.001428459770977497, -0.08556339889764786, -0.0952070876955986, 0.041995689272880554, -0.1202535480260849, -0.01066769752651453, 0.043444711714982986, -0.04269992187619209, 0.12283974885940552, -0.040432173758745193, -0.007321997079998255, 0.055041611194610596, 0.003923223819583654, -0.06601600348949432, -0.0031789555214345455, -0.015608920715749264, 0.013913601636886597, 0.007402278017252684, 0.014218656346201897, -0.08461038023233414, 0.02698861062526703, -0.04228398576378822, -0.02136833965778351, 0.015536555089056492, 0.04961182549595833, 0.07363967597484589, -0.10466285794973373, 0.005210317671298981, 0.07844141870737076, -0.033175189048051834, -0.1322903037071228, -0.0868099182844162, -0.07534613460302353, -0.04554476961493492, -0.0164048932492733, -0.026506656780838966, 0.007949941791594028, -0.0182399433106184, -0.0517999492585659, -0.07252563536167145, 0.023599285632371902, 0.05206107348203659, -0.057303350418806076, -0.09666356444358826, 0.044495485723018646, 0.004883192013949156, -0.05511088669300079, 0.07372823357582092, -0.006772668566554785, 0.030569585040211678, 0.05252925306558609, 0.11124249547719955, 0.008853021077811718, -0.0019371907692402601, 0.03833448886871338, 0.037561409175395966, -0.015580684877932072, 0.03981165587902069, 0.049070168286561966, -0.006949519272893667, -0.015260118991136551, -0.0832832083106041, -0.06730788946151733, 0.03521919623017311, -0.03924863785505295, -0.02580232359468937, -0.05239849165081978, -0.015273735858500004, -0.0011428530560806394, -0.0840705931186676, 0.06744348257780075, 0.06217053905129433, 0.035872187465429306, -0.0127906808629632, -0.036415282636880875, 0.039919137954711914, -0.03077578917145729, -0.0368623323738575, -0.0354931466281414, 0.08335445821285248, -0.047571875154972076, 0.023561187088489532, -0.12263979017734528, 0.03724728524684906, -0.06913828104734421, -0.04302017018198967, 0.0026693479157984257, -0.03369182348251343, -0.0255685243755579, -0.10069906711578369, -0.013446783646941185, 0.0481596440076828, -0.044240694493055344, 0.025013798847794533, 0.04790237545967102, -0.0463143065571785, -0.06570546329021454, 0.009200280532240868, -0.05090705305337906, 0.04666191712021828, 0.016258712857961655, 0.02338799461722374, -0.01469238381832838, -0.02688952162861824, -0.037063490599393845, -0.04772617295384407, 0.036033328622579575, 0.013595202006399632, -0.07186692208051682, -0.012121873907744884, 0.13698242604732513, -0.054272957146167755, 0.09579470753669739, -0.018000900745391846, 0.03515244647860527, 0.035747576504945755, 0.015358648262917995, -0.020873773843050003, -0.03420866280794144, 2.373642498165836e-33, 0.004087001550942659, 0.09396524727344513, 0.008987114764750004, 0.007892079651355743, 0.02842218615114689, 0.03252958133816719, -0.023905858397483826, 0.05588100850582123, -0.07259924709796906, -0.044402215629816055, 0.06960228830575943, -0.003998559899628162, 0.08624311536550522, 0.036521703004837036, 0.07360218465328217, -0.04819520562887192, 0.007478090934455395, 0.05898915231227875, -0.019743043929338455, -0.0237325020134449, -0.015354865230619907, -0.05497432500123978, 0.005782193969935179, 0.06376569718122482, 0.013976055197417736, 0.0641258955001831, 0.03309528902173042, 0.004411126486957073, 0.0591343455016613, -0.011793381534516811, -0.04374219477176666, 0.01053654495626688, 0.08095346391201019, 0.0012396350502967834, 0.06581103056669235, -0.0025814322289079428, -0.07841470092535019, -0.11471793055534363, -0.02738858386874199, 0.08644800633192062, -0.030907731503248215, 0.057290345430374146, -0.03507879748940468, -0.06745754182338715, -0.03848349303007126, 0.037257324904203415, 0.05201214179396629, 0.011970894411206245, -0.06959062069654465, 0.029294224455952644, -0.009705460630357265, 0.06902290880680084, -0.05948861688375473, 0.005885870195925236, 0.0029501882381737232, -0.03473135828971863, 0.05152532830834389, 0.007587505504488945, 0.056073691695928574, -0.08477016538381577, 0.012235607951879501, 0.023133469745516777, 0.001964848255738616, 0.046193916350603104, -0.028170393779873848, -0.0067735956981778145, -0.019663318991661072, 0.062182217836380005, -0.005953589919954538, 0.022341269999742508, 0.012766437605023384, 0.034293413162231445, 0.011621139012277126, -0.01770191825926304, 0.0020938110537827015, 0.021433182060718536, -0.016592945903539658, 0.03301786258816719, 0.021443868055939674, 0.05710439011454582, -0.049771927297115326, -0.028453068807721138, 0.014465024694800377, -0.007035637740045786, 0.045344434678554535, -0.049593571573495865, -0.04927557706832886, -0.0314156599342823, -0.12164126336574554, 0.04815191403031349, -0.03145309537649155, 0.01949147693812847, 0.02038230560719967, 0.06463321298360825, -0.016771676018834114, -2.316360658450266e-33, 0.007842582650482655, 0.034079741686582565, -0.03585878387093544, -0.0024113233666867018, 0.1221923902630806, -0.008060518652200699, 0.08166041225194931, -0.06345520168542862, 0.025417208671569824, 0.02269379235804081, -0.03757006302475929, 0.0024516331031918526, -0.010479165241122246, 0.03265863656997681, 0.047720059752464294, -0.001924882992170751, 0.060983091592788696, -0.12869161367416382, -0.07256944477558136, 0.008958404883742332, 0.14701144397258759, 0.05159146711230278, 0.006388494744896889, 0.02738032117486, -0.004137595649808645, -0.03915055841207504, 0.051600340753793716, -0.041472822427749634, 0.022494586184620857, 0.004335323814302683, 0.039017267525196075, -0.00763319618999958, -0.0055204397067427635, -0.01936323009431362, 0.05861395597457886, 0.06448338180780411, 0.06707989424467087, -0.003935203887522221, 0.014955636113882065, -0.06120843440294266, 0.1013767346739769, 0.050216082483530045, -0.015347855165600777, -0.034159790724515915, 0.011919219978153706, 0.05818180739879608, -0.08250715583562851, 0.052551671862602234, -0.09590791910886765, -0.02140170894563198, -0.019470779225230217, 0.021486302837729454, -0.08193112909793854, -0.08036662638187408, -0.09226694703102112, -0.07088422775268555, 0.006174936890602112, 0.03408947214484215, -0.0005518404068425298, -0.018194660544395447, 0.02761955000460148, -0.04814797639846802, 0.0399932935833931, -0.023555945605039597, 0.0016058675246313214, 0.036425698548555374, 0.0400175042450428, 0.11064691096544266, -0.0003956275468226522, 0.018695760518312454, 0.050038937479257584, -0.059090059250593185, 0.06259125471115112, 0.0765070840716362, 0.002242499962449074, 0.007463021669536829, 0.023093560710549355, -0.04647943750023842, 0.022281557321548462, -0.028783464804291725, 0.02034471184015274, 0.016215592622756958, 0.08534683287143707, 0.023307310417294502, 0.10140694677829742, 0.10804024338722229, -0.034638408571481705, 0.04119233787059784, -0.08908803015947342, 0.035973694175481796, -0.043563228100538254, 0.0729607418179512, -0.01990627683699131, 0.020396526902914047, -0.018512193113565445, -4.470294001635011e-08, -0.02850918285548687, -0.007024200167506933, -0.032094620168209076, -0.060947760939598083, -0.04950270801782608, -0.058581385761499405, 0.051795151084661484, 0.030354613438248634, 0.051092080771923065, -0.054129865020513535, -0.0013695991365239024, -0.10446909070014954, 0.019490908831357956, 0.08895456045866013, 0.0866464376449585, 0.028892185539007187, -0.044817015528678894, -0.029643280431628227, -0.031061984598636627, 0.03367877006530762, 0.03805943951010704, -0.025970734655857086, -0.0036888024769723415, 0.02999047376215458, -0.04606621712446213, 0.021130019798874855, 0.0333632156252861, 0.048508476465940475, -0.02722831629216671, -0.04446767270565033, -0.05041423439979553, 0.08033314347267151, -0.025748170912265778, -0.014920808374881744, 0.047536179423332214, -0.053025342524051666, -0.021161098033189774, -0.021410807967185974, 0.0544908307492733, 0.023105554282665253, -0.006709648296236992, 0.029420357197523117, -0.02852577157318592, 0.00629047304391861, 0.023637467995285988, 0.040019821375608444, 0.02754388563334942, -0.14894872903823853, -0.006592150777578354, 0.04156772792339325, -0.03889602795243263, -0.0013589700683951378, -0.03000982291996479, 0.0449916310608387, 0.08881128579378128, 0.04516582563519478, 0.13244342803955078, -0.008844813331961632, 0.0781707838177681, 0.03751348704099655, 0.0604221411049366, 0.017812488600611687, -0.029359012842178345, 0.02957269176840782]', 0: 4}])\n",
       "              .draw('full-hold');\n",
       "              });\n",
       "            }\n",
       "            document.head.appendChild(datatableScript);\n",
       "          };\n",
       "          document.head.appendChild(jqueryScript);\n",
       "        } else {\n",
       "          window.interactive_beam_jquery(document).ready(function($){\n",
       "            \n",
       "            var dt;\n",
       "            if ($.fn.dataTable.isDataTable(\"#table_df_08499c8cd95657156c076a29cd68a254\")) {\n",
       "              dt = $(\"#table_df_08499c8cd95657156c076a29cd68a254\").dataTable();\n",
       "            } else if ($(\"#table_df_08499c8cd95657156c076a29cd68a254_wrapper\").length == 0) {\n",
       "              dt = $(\"#table_df_08499c8cd95657156c076a29cd68a254\").dataTable({\n",
       "                \n",
       "            bAutoWidth: false,\n",
       "            columns: [{'title': ''}, {'title': 'id'}, {'title': 'content'}, {'title': 'title_and_content'}, {'title': 'metadata'}, {'title': 'embedding'}],\n",
       "            destroy: true,\n",
       "            responsive: true,\n",
       "            columnDefs: [\n",
       "              {\n",
       "                targets: \"_all\",\n",
       "                className: \"dt-left\"\n",
       "              },\n",
       "              {\n",
       "                \"targets\": 0,\n",
       "                \"width\": \"10px\",\n",
       "                \"title\": \"\"\n",
       "              }\n",
       "            ]\n",
       "              });\n",
       "            } else {\n",
       "              return;\n",
       "            }\n",
       "            dt.api()\n",
       "              .clear()\n",
       "              .rows.add([{1: '1_0', 2: 'Apache Beam is an open-source framework that provides a consistent programming model for both batch and streaming data processing. Developed originally by Google, it allows developers to write pipelines that can run on multiple engines, such as Apache Flink, Spark, and Google Cloud Dataflow. Beam uses abstractions like PCollections (data containers) and PTransforms (operations) to define the flow of data. The framework promotes portability through its runner architecture, letting the same pipeline execute on different backends. Support for multiple SDKs, including Java and Python, makes it accessible for a broad audience. Key features include support for event time, windowing, triggers, and stateful processing, which are essential for handling real-time data effectively. Beam is ideal for building ETL jobs, real-time analytics, and machine learning data pipelines. It helps teams focus on logic rather than infrastructure, offering flexibility and scalability in handling unbounded and bounded data sources. Apache Beam also supports a wide range of connectors for both input and output, including Kafka, BigQuery, and JDBC-based systems. This makes it easy to integrate Beam into existing data ecosystems. Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing.', 3: 'Apache Beam: Unified Model for Batch and Streaming Data. Apache Beam is an open-source framework that provides a consistent programming model for both batch and streaming data processing. Developed originally by Google, it allows developers to write pipelines that can run on multiple engines, such as Apache Flink, Spark, and Google Cloud Dataflow. Beam uses abstractions like PCollections (data containers) and PTransforms (operations) to define the flow of data. The framework promotes portability through its runner architecture, letting the same pipeline execute on different backends. Support for multiple SDKs, including Java and Python, makes it accessible for a broad audience. Key features include support for event time, windowing, triggers, and stateful processing, which are essential for handling real-time data effectively. Beam is ideal for building ETL jobs, real-time analytics, and machine learning data pipelines. It helps teams focus on logic rather than infrastructure, offering flexibility and scalability in handling unbounded and bounded data sources. Apache Beam also supports a wide range of connectors for both input and output, including Kafka, BigQuery, and JDBC-based systems. This makes it easy to integrate Beam into existing data ecosystems. Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing.', 4: \"{'title': 'Apache Beam: Unified Model for Batch and Streaming Data', 'keywords': ['Apache Beam', 'stream processing', 'batch processing', 'data pipelines', 'SDK'], 'tags': ['Data Engineering', 'Open Source', 'Streaming', 'Batch', 'Big Data']}\", 5: '[-0.06466388702392578, -0.029533877968788147, -0.04261693358421326, 0.01254373136907816, 0.002130179898813367, -0.08420056104660034, -0.035637639462947845, -0.016679927706718445, -0.05082850158214569, -0.08004148304462433, -0.071900874376297, 0.024430863559246063, -0.04954373463988304, -0.048748258501291275, -0.011967036873102188, 0.006303164642304182, 0.05958455801010132, 0.0028353261295706034, -0.07481587678194046, -0.11607144773006439, -0.03259394317865372, -0.041554611176252365, -0.03504106029868126, -0.001439800951629877, -0.02360020950436592, 0.059943076223134995, -0.01519234012812376, -0.034146957099437714, 0.004027337301522493, -0.054018571972846985, 0.02313961647450924, -0.031018856912851334, -0.05701017752289772, 0.040523823350667953, -0.04349478334188461, 0.021394044160842896, 0.007265422493219376, -0.04372619464993477, -0.12264154851436615, -0.09200466424226761, 0.03763662651181221, -0.05324647203087807, -0.09666550904512405, 0.022716403007507324, -0.08052070438861847, -0.07522332668304443, -0.01606123149394989, -0.11872679740190506, -0.008652692660689354, 0.07510612159967422, -0.08550882339477539, -0.06455493718385696, 0.009230186231434345, 0.07337071001529694, -0.007284300867468119, 0.053089335560798645, 0.025012554600834846, 0.00547427823767066, -0.011389803141355515, 0.02432127669453621, -0.024486027657985687, -0.04197389632463455, -0.00921629648655653, 0.05588779225945473, -0.04310512915253639, -0.05197839438915253, 0.06383068859577179, 0.051967501640319824, 0.1240178570151329, -0.10710014402866364, -0.056561537086963654, 0.06153789162635803, -0.01450709905475378, 0.023290887475013733, -0.0773656889796257, 0.0013129215221852064, 0.0134525615721941, -0.002708431100472808, 0.0018889567581936717, 0.007732840720564127, 0.00323132099583745, 0.018673783168196678, -0.09686124324798584, 0.009700699709355831, 0.023337548598647118, -0.0077779777348041534, -0.013852821663022041, 0.11304262280464172, -0.03682168573141098, 0.026285415515303612, -0.0008936264785006642, 0.02425266243517399, 0.07036064565181732, -0.014448655769228935, -0.004841167479753494, 0.03809678927063942, 0.02485431730747223, -0.035138584673404694, -0.0029179861303418875, 0.04480774700641632, -0.030326586216688156, 0.045793332159519196, 0.05787701532244682, -0.0073225838132202625, -0.02137342281639576, -0.08531729876995087, -0.10222272574901581, 0.1223769336938858, -0.02004469744861126, -0.07306617498397827, 0.0631437674164772, -0.008705795742571354, -0.002412058413028717, -0.05419433116912842, 0.038417965173721313, -0.0283599141985178, -0.09585224837064743, -0.0484754703938961, -0.013657975010573864, 0.03131190687417984, 0.031260404735803604, 0.017582444474101067, -0.013743625022470951, 0.013575058430433273, 0.06819900870323181, 0.009789698757231236, -0.08053038269281387, 4.226467530520472e-33, 0.0030900901183485985, -0.015385894104838371, -0.042228877544403076, -0.07484018802642822, 0.05486077442765236, -0.03897050768136978, 0.044820234179496765, 0.031497374176979065, -0.09605666995048523, -0.027088796719908714, 0.03788488730788231, 0.06610210984945297, 0.05553589016199112, -0.004346764646470547, 0.06688157469034195, -0.0606110654771328, -0.042873118072748184, 0.07787125557661057, 0.06925395131111145, 0.01790660433471203, 0.05190691724419594, -0.049133073538541794, -0.028836533427238464, 0.03326796367764473, 0.016870610415935516, 0.03505547344684601, 0.032689593732357025, 0.02542627975344658, -0.025435226038098335, 0.04890316352248192, 0.002155381953343749, -0.020533351227641106, -0.003533762414008379, 0.0411994643509388, 0.055960118770599365, -0.07614350318908691, -0.054680973291397095, -0.09691374748945236, 0.009100464172661304, -0.010418709367513657, -0.05354749411344528, 0.0027101878076791763, -0.03828197717666626, -0.01852724887430668, -0.05890074372291565, -0.044915273785591125, -0.013576041907072067, 0.035576049238443375, -0.04080145061016083, 0.005122833885252476, 0.09771314263343811, -0.003387728938832879, 0.11922183632850647, 0.04530109465122223, 0.05175572261214256, 0.057451214641332626, 0.02983781509101391, 0.013441174291074276, 0.04309932142496109, 0.05667578801512718, -0.0833202451467514, -0.0006599651533178985, -0.053148698061704636, 0.03123246692121029, 0.01695791445672512, -0.02130052074790001, 0.01621287688612938, 0.014019771479070187, 0.005802274215966463, 0.030075233429670334, 0.007117005065083504, 0.04007411375641823, 0.04097031056880951, 0.07147867977619171, 0.03629567474126816, 0.0022272050846368074, 0.022832626476883888, 0.002338982652872801, -0.050566937774419785, 0.03600058704614639, -0.05889451876282692, -0.0052792844362556934, 0.03501333296298981, 0.004061616957187653, 0.036560703068971634, -0.0014219945296645164, -0.057203203439712524, 0.033465780317783356, -0.07556362450122833, 0.01921098865568638, -0.024971656501293182, 0.05516916885972023, 0.07144436985254288, 0.00015671631263103336, -0.01950802654027939, -3.650351137026746e-33, 0.017354466021060944, 0.059810806065797806, -0.05945239216089249, 0.06891019642353058, 0.11440429836511612, 0.009432020597159863, 0.009542316198348999, -0.04723283648490906, 0.005809252616018057, -0.03874228522181511, -0.078343965113163, -0.013072866015136242, 0.008306140080094337, 0.0019391959067434072, 0.0013291530776768923, -0.016417009755969048, -0.00790349580347538, -0.1526392549276352, -0.014709247276186943, 0.0038721419405192137, -0.01859547197818756, 0.05286412686109543, -0.02104310132563114, 0.016759684309363365, -0.012588641606271267, -0.04436327889561653, -0.061210691928863525, -0.05035687983036041, -0.02498915046453476, 0.0033777521457523108, 0.024367185309529305, -0.018248409032821655, 0.025040041655302048, -0.10882879048585892, 0.006693651434034109, -0.005851115100085735, 0.1111128181219101, 0.007033593021333218, 0.03136799484491348, 0.06491357088088989, 0.09991703927516937, 0.013178274035453796, -0.02550503984093666, -0.009967111982405186, -0.054682184010744095, 0.09739664196968079, -0.05937360227108002, 0.11696966737508774, -0.034573908895254135, -0.021528026089072227, -0.10162897408008575, 0.04898342490196228, -0.019914917647838593, 0.0065137529745697975, 0.03217782452702522, -0.06474792957305908, 0.07378212362527847, 0.00545730022713542, -0.10361640155315399, 0.0034713854547590017, 0.01891847513616085, -0.07079721987247467, 0.07830165326595306, 0.02078700065612793, 0.01653195545077324, -0.01404052134603262, 0.021191304549574852, -0.011555657722055912, -0.15100634098052979, -0.02106490358710289, 0.07326526194810867, -0.09215068072080612, 0.013960559852421284, 0.09145322442054749, 0.0014025433920323849, -0.04531499743461609, -0.00909416563808918, -0.03184480965137482, 0.041091516613960266, 0.12100711464881897, 0.024355394765734673, 0.07038372755050659, 0.0430283360183239, 0.03169526159763336, 0.10590188205242157, 0.027500400319695473, -0.0008741550846025348, -0.015190372243523598, 0.0063900393433868885, -0.01886691153049469, -0.05316048115491867, 0.08248165994882584, -0.06078287959098816, 0.06999880820512772, 0.054216716438531876, -4.799430541879701e-08, 0.024673016741871834, 0.02466781623661518, -0.08306656777858734, 0.00029196596005931497, -0.0018005361780524254, 0.005545003805309534, 0.02234015241265297, 0.13406866788864136, 0.04606040194630623, -0.012594856321811676, 0.0772823765873909, -0.07090407609939575, -0.04751046374440193, 0.022145597264170647, 0.07882999628782272, 0.06339012086391449, 0.06228167191147804, -0.040852054953575134, -0.035441990941762924, -0.02966366894543171, 0.04845830798149109, -0.000818374683149159, -0.00830867700278759, 0.05047774314880371, -0.051014143973588943, 0.009566603228449821, 0.12313132733106613, 0.03784331679344177, 0.02365674078464508, -0.06346344202756882, -0.054321423172950745, -0.01641339249908924, 0.0401403084397316, 0.023231539875268936, 0.03016555868089199, 0.03899690881371498, 0.015127046965062618, 0.01934240385890007, 0.02179247885942459, 0.06229304149746895, 0.001949132769368589, 0.05174288898706436, -0.051228415220975876, -0.008371411822736263, -0.022170983254909515, 0.03172885254025459, -0.04765719547867775, -0.031184902414679527, -0.029825732111930847, 0.052426744252443314, 0.011883549392223358, -0.04567771404981613, 0.005567905493080616, 0.022183997556567192, 0.10563989728689194, 0.06372497975826263, 0.04733740910887718, -0.08152730017900467, 0.08148864656686783, 0.059733033180236816, 0.036120910197496414, 0.034422941505908966, 0.03181681036949158, 0.0001891597785288468]', 0: 0}, {1: '1_1', 2: \"Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing. The concept of runners enables developers to write once and run anywhere, which is particularly appealing for organizations that want to avoid vendor lock-in. The Beam model is based on a unified programming model that decouples pipeline logic from execution. This makes it easier to reason about time and state in both batch and streaming pipelines. Advanced features like late data handling, watermarks, and session windowing allow for more accurate and meaningful processing of real-world data. Beam also integrates with orchestration tools and monitoring systems, allowing for production-grade deployments. Community support and contributions have grown significantly, making Beam a stable and evolving ecosystem. Many cloud providers offer native support for Beam pipelines, and it's increasingly a core component in modern data platform architectures.\", 3: \"Apache Beam: Unified Model for Batch and Streaming Data. Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing. The concept of runners enables developers to write once and run anywhere, which is particularly appealing for organizations that want to avoid vendor lock-in. The Beam model is based on a unified programming model that decouples pipeline logic from execution. This makes it easier to reason about time and state in both batch and streaming pipelines. Advanced features like late data handling, watermarks, and session windowing allow for more accurate and meaningful processing of real-world data. Beam also integrates with orchestration tools and monitoring systems, allowing for production-grade deployments. Community support and contributions have grown significantly, making Beam a stable and evolving ecosystem. Many cloud providers offer native support for Beam pipelines, and it's increasingly a core component in modern data platform architectures.\", 4: \"{'title': 'Apache Beam: Unified Model for Batch and Streaming Data', 'keywords': ['Apache Beam', 'stream processing', 'batch processing', 'data pipelines', 'SDK'], 'tags': ['Data Engineering', 'Open Source', 'Streaming', 'Batch', 'Big Data']}\", 5: '[-0.018973594531416893, -0.038047756999731064, 0.009853314608335495, 0.017600156366825104, 0.01497685257345438, -0.06582952290773392, -0.07951309531927109, -0.02883726730942726, -0.041912175714969635, -0.04387732222676277, -0.08884589374065399, 0.053467925637960434, -0.05935873091220856, -0.03815134987235069, 0.01334142591804266, -0.0048108333721756935, 0.0428265780210495, -0.037232715636491776, -0.11089688539505005, -0.08068843185901642, -0.05602698773145676, -0.06249658390879631, -0.09883567690849304, 0.05031457543373108, -0.020239757373929024, 0.07551726698875427, -0.06086629256606102, -0.05752619728446007, 0.01581764407455921, -0.04817376285791397, -0.0023335570003837347, -0.04845050722360611, -0.0006991951959207654, 0.06703019887208939, -0.0055598048493266106, 0.016426336020231247, 0.02817388065159321, -0.07312498986721039, -0.07833398133516312, -0.06406418979167938, 0.028878629207611084, -0.0944715365767479, -0.09821832925081253, 0.044137436896562576, -0.009284254163503647, -0.07901174575090408, -0.04781392216682434, -0.09630963951349258, -0.07280350476503372, 0.08982539176940918, -0.04611873999238014, -0.08185379952192307, 0.02143690548837185, 0.035766519606113434, -0.029929867014288902, 0.0987226590514183, 0.053078461438417435, 0.06040719151496887, 0.0037211780436336994, 0.01677882857620716, -0.0024818717502057552, -0.06774415075778961, -0.022503690794110298, 0.047713641077280045, 0.03527110442519188, -0.033364977687597275, 0.009873880073428154, 0.04558619111776352, 0.11955580115318298, -0.09005599468946457, -0.010989201255142689, 0.031770385801792145, 0.008936704136431217, 0.00538625055924058, -0.06366196274757385, 0.03144121542572975, 0.01911686360836029, -0.04297826811671257, -0.012642892077565193, -0.007212298922240734, 0.03595840930938721, -0.018883714452385902, -0.06322586536407471, 0.025581810623407364, 0.02180952951312065, 0.06087557598948479, 0.04372349753975868, 0.057813845574855804, 0.01641206257045269, -0.026707131415605545, -0.050563156604766846, 0.004152100067585707, 0.09263436496257782, -0.015737539157271385, -0.0007108922582119703, 0.02848093770444393, -0.00806748028844595, -0.05122291296720505, 0.005889056716114283, 0.04067201167345047, -0.00850051362067461, 0.02599603682756424, 0.10257657617330551, 0.01619791053235531, 0.00833812728524208, -0.08265282213687897, -0.04061337560415268, 0.1259007751941681, -0.027060644701123238, -0.06887192279100418, 0.04976821318268776, -0.015719763934612274, 0.023927684873342514, -0.036306049674749374, 0.05325644835829735, -0.06689215451478958, -0.09015952050685883, -0.008463169448077679, -0.03694949671626091, 0.06418510526418686, 0.04649828001856804, -0.004484950564801693, -0.011765721254050732, -0.03851206228137016, 0.06064915657043457, 0.029278069734573364, -0.05129281431436539, 3.0011401170759893e-33, 0.005440344102680683, -0.04820927232503891, -0.03920764848589897, -0.043692316859960556, 0.11088904738426208, -0.04808446019887924, 0.049140360206365585, 0.011111056432127953, -0.10306145995855331, -0.02553553320467472, 0.05190734192728996, 0.05066768079996109, 0.04280221089720726, 0.03775731474161148, 0.04095909371972084, -0.08128233999013901, -0.022233635187149048, 0.073038749396801, 0.0646248310804367, 0.04169369861483574, 0.06933081150054932, -0.05534159019589424, -0.03268785774707794, -0.005426599644124508, 0.06439580768346786, -0.01288049016147852, 0.04870288819074631, 0.026471689343452454, -0.005027782171964645, 0.02150837890803814, 0.005291462875902653, 0.0027808905579149723, 0.03742994740605354, 0.01261330209672451, 0.06765583157539368, -0.03537099435925484, -0.017706256359815598, -0.09280066192150116, 0.0316665880382061, 0.0168000478297472, -0.029039543122053146, 0.015987912192940712, -0.07633719593286514, -0.024627406150102615, -0.040600694715976715, -0.027892230078577995, -0.033759672194719315, -0.002126255538314581, -0.011880690231919289, -0.028114158660173416, 0.08075553923845291, 0.06590402126312256, 0.04470798373222351, 0.04803197458386421, 0.07680407166481018, -0.03015856444835663, 0.05029886215925217, -0.032047562301158905, 0.07621696591377258, 0.03183460608124733, -0.031034624204039574, -0.024675123393535614, -0.07854422181844711, 0.01856403425335884, -0.007004621904343367, -0.03843999654054642, 0.047416381537914276, 0.0338081531226635, 0.026617716997861862, 0.03181052953004837, -0.03686947375535965, -0.01137223094701767, -0.02098255045711994, 0.0560978427529335, 0.032936133444309235, -0.017804542556405067, -0.012492518872022629, 0.020227478817105293, -0.02718948945403099, 0.015233292244374752, -0.03476536273956299, -0.03465321287512779, -0.009879112243652344, 0.011551113799214363, 0.04184458777308464, -0.0032065215054899454, -0.0724199116230011, -0.002717012306675315, -0.11437557637691498, -0.0005933393258601427, -0.009862713515758514, 0.042416442185640335, 0.04581250995397568, 0.01821465790271759, 0.017902765423059464, -2.764706693481364e-33, 0.0022231105249375105, 0.014193677343428135, -0.07299257069826126, 0.08803818374872208, 0.08409763872623444, -0.03365476429462433, -0.0013658803654834628, -0.0987565815448761, -0.0033523032907396555, -0.02895462140440941, -0.09524943679571152, -0.0017201234586536884, -0.03854745253920555, 0.03291812166571617, -0.03718029335141182, -0.018591750413179398, -0.005976638291031122, -0.12552377581596375, -0.006126183085143566, -0.002084973966702819, 0.0248258113861084, 0.03312137722969055, -0.04124243184924126, 0.009689252823591232, -0.011249818839132786, -0.0053729587234556675, -0.0344584584236145, -0.058878734707832336, 0.017762307077646255, -0.052064865827560425, 0.0032393925357609987, 0.002994223264977336, 0.020968977361917496, -0.1437208503484726, -0.0004785321361850947, 0.030647477135062218, 0.0286928191781044, 0.044096317142248154, 0.019514000043272972, -0.0012711107265204191, 0.06709816306829453, 0.01644347794353962, 0.02145499736070633, -0.03152571991086006, -0.04268411546945572, 0.1273767203092575, 0.02085105888545513, 0.051103588193655014, -0.09107273072004318, -0.022750038653612137, -0.03109086863696575, -0.01736641116440296, 0.0010191221954301, 0.0037347853649407625, 0.03477935120463371, -0.08281423896551132, 0.05154472216963768, -0.026259252801537514, -0.11400901526212692, 0.025394976139068604, 0.04734973981976509, -0.03389143571257591, 0.12601801753044128, -0.00430592754855752, -0.024579815566539764, -0.0002861841639969498, 0.017392922192811966, -0.013216436840593815, -0.11752787232398987, 0.012718579731881618, 0.013284187763929367, -0.08756081759929657, -0.03979286178946495, 0.09630093723535538, -0.033030349761247635, -0.06131284683942795, -0.03974708914756775, -0.05123161897063255, -0.004189329221844673, 0.09414182603359222, -0.06169441342353821, 0.05233728513121605, 0.032566361129283905, 0.014203887432813644, 0.09891701489686966, 0.04159114882349968, 0.04394884407520294, 0.002195159438997507, -0.0052054510451853275, 0.034669600427150726, -0.057895563542842865, 0.09207107126712799, -0.07015221565961838, 0.05211507901549339, -0.02817714586853981, -4.9893632336761584e-08, -0.00896522868424654, 0.04456833750009537, -0.0719098225235939, 0.04353230446577072, 0.0015314699849113822, -0.010044410824775696, 0.06597968190908432, 0.09209451824426651, 0.09022504091262817, -0.010507077910006046, 0.07367205619812012, -0.09957010298967361, -0.028277182951569557, 0.0457858182489872, 0.12363816797733307, 0.04674151912331581, 0.04391893744468689, 0.0015561962500214577, -0.0487351231276989, -0.04268272593617439, 0.006729984190315008, 0.04273779317736626, 0.0370924174785614, 0.07646933197975159, -0.016536453738808632, 0.030621210113167763, 0.09810485690832138, 0.06789854168891907, 0.019589059054851532, -0.003846997395157814, -0.016470609232783318, 0.0031430134549736977, 0.06656448543071747, 0.009443582966923714, 0.06827470660209656, 0.04586328566074371, 0.03099874034523964, 0.01014702022075653, 0.01989901438355446, 0.030771005898714066, -0.0008424646803177893, 0.03357389569282532, 0.012780584394931793, 0.019024720415472984, -0.003248361637815833, 0.0033370780292898417, -0.059965137392282486, -0.004392987582832575, -0.04737864434719086, -0.006688870955258608, 0.01745840534567833, -0.06501461565494537, -0.0549679696559906, 0.07742945104837418, 0.1438921093940735, 0.05695037171244621, 0.06452634185552597, -0.06825384497642517, 0.0938495546579361, 0.08684926480054855, 0.00600523641332984, -0.02143688313663006, 0.046269144862890244, -0.03446299210190773]', 0: 1}, {1: '2_0', 2: 'Google Cloud Dataflow is a fully managed service that runs Apache Beam pipelines in the cloud. It abstracts away infrastructure management and handles dynamic scaling, load balancing, and fault tolerance. Developers can focus on writing data logic using the Beam SDK and deploy it easily to Google Cloud. Dataflow supports both batch and stream processing and integrates seamlessly with other Google services like BigQuery, Pub/Sub, and Cloud Storage. Its autoscaling capabilities allow it to adapt to changing data volumes, optimizing for cost and performance. Features like monitoring dashboards, job templates, and built-in logging make it suitable for both development and production use. With support for event time processing, stateful functions, and windowing, Dataflow is well-suited for real-time analytics and data transformation tasks. It’s a key component for architects building scalable, cloud-native data platforms. Dataflow also offers templates for common ETL tasks, helping teams get started quickly with minimal setup. Its integration with Cloud Functions and Cloud Composer enables event-driven and orchestrated workflows. Security and compliance are built-in with IAM roles, encryption at rest and in transit, and audit logging, making it suitable for enterprise environments.', 3: 'Google Cloud Dataflow: Run Apache Beam in the Cloud. Google Cloud Dataflow is a fully managed service that runs Apache Beam pipelines in the cloud. It abstracts away infrastructure management and handles dynamic scaling, load balancing, and fault tolerance. Developers can focus on writing data logic using the Beam SDK and deploy it easily to Google Cloud. Dataflow supports both batch and stream processing and integrates seamlessly with other Google services like BigQuery, Pub/Sub, and Cloud Storage. Its autoscaling capabilities allow it to adapt to changing data volumes, optimizing for cost and performance. Features like monitoring dashboards, job templates, and built-in logging make it suitable for both development and production use. With support for event time processing, stateful functions, and windowing, Dataflow is well-suited for real-time analytics and data transformation tasks. It’s a key component for architects building scalable, cloud-native data platforms. Dataflow also offers templates for common ETL tasks, helping teams get started quickly with minimal setup. Its integration with Cloud Functions and Cloud Composer enables event-driven and orchestrated workflows. Security and compliance are built-in with IAM roles, encryption at rest and in transit, and audit logging, making it suitable for enterprise environments.', 4: \"{'title': 'Google Cloud Dataflow: Run Apache Beam in the Cloud', 'keywords': ['Google Cloud', 'Dataflow', 'Apache Beam', 'serverless', 'stream and batch'], 'tags': ['Cloud Computing', 'Data Pipelines', 'Google Cloud', 'Serverless', 'Enterprise']}\", 5: '[-0.06922302395105362, 0.023161506280303, 0.045865464955568314, -0.043821707367897034, 0.012096977792680264, -0.05220700427889824, 0.001474533462896943, -0.060477327555418015, 0.02311597764492035, 0.006155427545309067, -0.09254368394613266, -0.026979118585586548, 0.007794504053890705, -0.03784167021512985, -0.008246206678450108, -0.014848004095256329, 0.06783697754144669, -0.033511579036712646, -0.11154574900865555, -0.08980037271976471, -0.06648188084363937, -0.020473342388868332, -0.07527496665716171, 0.03519774600863457, -0.038868505507707596, 0.0657028779387474, -0.014987119473516941, -0.08130184561014175, 0.009730864316225052, -0.05540500208735466, -0.013050466775894165, -0.04525986686348915, -0.004191172309219837, 0.13747315108776093, -0.04854223132133484, -0.010875235311686993, 0.0329212062060833, -0.011579407379031181, -0.07976572215557098, -0.07062806934118271, 0.04519709199666977, -0.1190754696726799, -0.0873320996761322, -0.05216065049171448, -0.051656756550073624, -0.03193237632513046, -0.029147827997803688, -0.12397291511297226, -0.049637433141469955, 0.0897335410118103, -0.02015087939798832, -0.0870516300201416, 0.0478004589676857, 0.061090752482414246, -0.07045301049947739, 0.06690108776092529, 0.05036143586039543, -0.006564647890627384, 0.028151128441095352, 0.0011042800033465028, -0.03730063512921333, -0.04508212208747864, -0.024164803326129913, 0.03232676535844803, -0.03507281094789505, 0.0007927772821858525, 0.06290047615766525, 0.013078553602099419, 0.0635475292801857, -0.0221528559923172, 0.0028895079158246517, 0.02719644084572792, -0.03433110564947128, -0.015487424097955227, -0.07674422860145569, -0.04146977514028549, -0.044433802366256714, -0.04066569358110428, -0.0005891985492780805, -0.02918723039329052, 0.04124194011092186, 0.06517095863819122, -0.030084604397416115, 0.08247551321983337, -0.021744975820183754, -0.06545085459947586, -0.015583222731947899, 0.07846656441688538, 0.017796527594327927, -0.0050199697725474834, -0.012236016802489758, 0.040639594197273254, 0.04380558431148529, -0.03247296065092087, 0.07249102741479874, 0.023049896582961082, -0.04953192174434662, -0.05010318011045456, 0.05910531431436539, -0.01007019355893135, -0.012454433366656303, 0.09351649135351181, 0.05062120407819748, -0.017186110839247704, 0.03257777541875839, -0.036310818046331406, -0.07265055179595947, 0.085996612906456, 0.0035129461903125048, -0.02146696299314499, 0.04926305636763573, 0.06111319735646248, 0.016581900417804718, -0.027423610910773277, 0.06676501780748367, 0.019711701199412346, -0.07355605065822601, -0.05667809024453163, -0.07576645165681839, 0.10831525176763535, 0.0639774426817894, 0.019096162170171738, -0.045362140983343124, 0.023760955780744553, 0.052066754549741745, 0.0030979060102254152, -0.03407223895192146, 2.9031551324242667e-33, 0.030751287937164307, -0.01548618171364069, -0.0002451580367051065, -0.06648711115121841, 0.10106846690177917, 0.022745406255126, -0.0013853602577000856, -0.010138805955648422, -0.061261966824531555, -0.016061361879110336, -0.02694455347955227, 0.10520829260349274, -0.013813639059662819, 0.006083263549953699, 0.05271698161959648, -0.01239310298115015, -0.031855691224336624, 0.09228593111038208, 0.09149166941642761, 0.02157820574939251, 0.025037162005901337, -0.09339465945959091, 0.0025386337656527758, 0.03986256942152977, 0.06561947613954544, 0.003864482045173645, 0.07566480338573456, -0.005065985955297947, 0.05211842060089111, 0.01600526086986065, -0.03600906580686569, -0.04902055859565735, 0.01332682091742754, 0.047950517386198044, -0.02850937284529209, -0.04355623200535774, -0.10912280529737473, -0.04597802832722664, 0.04209766909480095, 0.037293560802936554, -0.003396362764760852, 0.013617961667478085, -0.07035071402788162, -0.05462810769677162, -0.0700947716832161, 0.015253187157213688, -0.008539300411939621, -0.009663798846304417, 0.04764934629201889, -0.006874932907521725, 0.12136242538690567, 0.017473505809903145, 0.03491348400712013, -0.006369094364345074, 0.06318545341491699, 0.030066514387726784, 0.06774647533893585, -0.023543784394860268, 0.001241658697836101, 0.05070498585700989, -0.06009829044342041, -0.01978699490427971, -0.03339686989784241, 0.05812661722302437, 0.00752455135807395, -0.083124078810215, 0.014432292431592941, 0.09036456793546677, 0.014141008257865906, -0.013721432536840439, -0.028522038832306862, -0.006297665182501078, 0.08145670592784882, 0.07038164883852005, 0.0544198676943779, -0.008289210498332977, -0.007178945001214743, 0.015230788849294186, -0.07144331187009811, 0.04001161456108093, -0.047253143042325974, -0.017992176115512848, 0.053590234369039536, 0.0423310287296772, 0.0031884820200502872, -0.03413547948002815, -0.014060970395803452, 0.04878852143883705, -0.12464924901723862, 0.019769102334976196, -0.05688019469380379, -0.019350357353687286, 0.10566561669111252, 0.0054002427496016026, -0.05828821286559105, -3.197660181001421e-33, -0.015062433667480946, 0.005063250195235014, -0.03854287415742874, 0.09942492842674255, 0.1009494811296463, 0.007028763648122549, 0.0347612090408802, 0.001351911574602127, 0.01721402257680893, 0.02427109330892563, -0.05357150733470917, 0.007844974286854267, 0.004905079957097769, -0.00778529979288578, -0.007310265675187111, -0.03678283840417862, -0.009034755639731884, -0.11026682704687119, -0.052736155688762665, -0.02109014242887497, -0.018955320119857788, 0.01524350605905056, -0.017219502478837967, -0.01637597195804119, -0.024668259546160698, 0.005761103704571724, -0.061775751411914825, -0.02436325140297413, 0.019888758659362793, 0.012886084616184235, -0.031120268628001213, 0.034060172736644745, 0.058737706393003464, -0.07428651303052902, 0.06617841124534607, -0.07656872272491455, 0.06839325278997421, 0.03815082460641861, 0.017387909814715385, -0.008589591830968857, -0.012417268939316273, -0.007639465853571892, 0.03485056012868881, -0.0672459825873375, -0.036845918744802475, 0.10244598239660263, 0.00019573078316170722, 0.06977578997612, -0.0555693581700325, -0.05785490944981575, -0.0764404758810997, 0.032553814351558685, -0.03484220430254936, 0.033937014639377594, 0.009329676628112793, -0.0365307480096817, 0.020915787667036057, -0.021535539999604225, -0.11950027197599411, 0.0016304069431498647, -0.01581830158829689, -0.013064299710094929, 0.12702429294586182, 0.01419518981128931, -0.023599980399012566, 0.02876134403049946, 0.0182856023311615, -0.002988226478919387, -0.13134202361106873, 0.03873820602893829, 0.05212586373090744, -0.08891689777374268, -0.0006976487347856164, 0.05933821201324463, 0.002799754962325096, -0.023531176149845123, 0.031348053365945816, -0.13147170841693878, 0.03059895522892475, 0.030061455443501472, 0.01293276622891426, 0.05505232885479927, -0.010720016434788704, 0.021187424659729004, 0.06960820406675339, -0.002950818045064807, 0.005676741246134043, -0.027272380888462067, 0.04858771711587906, 0.0734899491071701, -0.1088961735367775, 0.00760533194988966, -0.11584514379501343, 0.029936376959085464, 0.01612277142703533, -4.575245071691825e-08, -0.01872018165886402, 0.047710951417684555, -0.04508286714553833, 0.000703521363902837, 0.007688414771109819, -0.031201617792248726, 0.05539138987660408, 0.11052795499563217, 0.04336031526327133, -0.009162266738712788, 0.04374673217535019, -0.11823780834674835, -0.01510899793356657, -0.018336022272706032, 0.05745872110128403, 0.002433251589536667, 0.07766743004322052, -0.04140744358301163, -0.02027367614209652, -0.05821099132299423, -0.00770132802426815, 0.030263029038906097, -0.01441414374858141, 0.024740438908338547, -0.02200401946902275, 0.00046951076365076005, 0.09944584965705872, 0.013641657307744026, 0.013016611337661743, -0.06615390628576279, -0.040024783462285995, 0.0012006880715489388, 0.035945646464824677, 0.0238544549793005, 0.04967077821493149, -0.03130926564335823, 0.005723670590668917, -0.02381395734846592, 0.03850293904542923, 0.0029672004748135805, 0.04136791080236435, 0.05762109532952309, -0.015518826432526112, -0.025523079559206963, -0.012718209996819496, -0.016003094613552094, -0.03060447797179222, 0.0014687910443171859, -0.00880576390773058, 0.115822434425354, 0.01786453276872635, -0.045547958463430405, 0.02170303650200367, 0.09815438836812973, 0.10668042302131653, 0.034177426248788834, 0.03066622093319893, -0.10539556294679642, 0.12173125892877579, 0.0512351468205452, 0.03973713889718056, 0.030443686991930008, 0.007508042734116316, -0.03698401898145676]', 0: 2}, {1: '2_1', 2: 'For developers, Dataflow provides local testing capabilities and a unified logging system through Cloud Logging. It also supports SQL-based pipeline definitions using BigQuery, which lowers the barrier to entry for analysts and data engineers. Dataflow’s streaming engine significantly improves performance and reduces costs by decoupling compute and state management. In summary, Google Cloud Dataflow not only simplifies the deployment of Apache Beam pipelines but also enhances them with cloud-native features. Its managed runtime, high availability, and integration with the broader Google Cloud ecosystem make it a powerful tool for modern data processing.', 3: 'Google Cloud Dataflow: Run Apache Beam in the Cloud. For developers, Dataflow provides local testing capabilities and a unified logging system through Cloud Logging. It also supports SQL-based pipeline definitions using BigQuery, which lowers the barrier to entry for analysts and data engineers. Dataflow’s streaming engine significantly improves performance and reduces costs by decoupling compute and state management. In summary, Google Cloud Dataflow not only simplifies the deployment of Apache Beam pipelines but also enhances them with cloud-native features. Its managed runtime, high availability, and integration with the broader Google Cloud ecosystem make it a powerful tool for modern data processing.', 4: \"{'title': 'Google Cloud Dataflow: Run Apache Beam in the Cloud', 'keywords': ['Google Cloud', 'Dataflow', 'Apache Beam', 'serverless', 'stream and batch'], 'tags': ['Cloud Computing', 'Data Pipelines', 'Google Cloud', 'Serverless', 'Enterprise']}\", 5: '[-0.03661690652370453, -0.03150395676493645, 0.06368837505578995, -0.03153130039572716, -0.004471856635063887, -0.09541423618793488, -0.054404497146606445, -0.09089410305023193, -0.050178591161966324, -0.006378731224685907, -0.06700551509857178, -0.0004740693257190287, -0.008150320500135422, -0.07737687230110168, -0.027979711070656776, 0.0010401720646768808, 0.046904824674129486, -0.0015425728633999825, -0.08718173205852509, -0.08254003524780273, -0.04283147677779198, 0.00358956097625196, -0.07748213410377502, 0.04272042214870453, 0.035424426198005676, 0.06755490601062775, -0.05363599956035614, -0.058646537363529205, 0.033957578241825104, -0.02617446333169937, -0.020357120782136917, -0.007250271271914244, -0.059132955968379974, 0.13508857786655426, -0.021537745371460915, -0.026470733806490898, 0.0009676427580416203, 0.008185886777937412, -0.07078035175800323, -0.04654406011104584, 0.07878093421459198, -0.1012917160987854, -0.04706841707229614, -0.035616252571344376, -0.0722387358546257, -0.022064778953790665, -0.00020241140737198293, -0.10148043185472488, -0.03747682273387909, 0.06451018899679184, -0.053426895290613174, -0.07141502946615219, -0.005222890060395002, 0.09770642220973969, -0.04477661848068237, 0.03803936019539833, 0.03609953820705414, 0.011689776554703712, 0.010280258022248745, 0.05735364183783531, -0.048317525535821915, -0.02284780703485012, 0.012211902067065239, 0.0429220050573349, -0.021692665293812752, -0.020572056993842125, 0.07219453901052475, 0.031036250293254852, 0.0844205915927887, -0.06730155646800995, -0.009439797140657902, 0.08722429722547531, -0.05560028925538063, -0.00539100356400013, -0.0876370370388031, -0.009375905618071556, -0.012034732848405838, 0.0021236685570329428, 0.020318949595093727, 0.024571334943175316, 0.018642617389559746, 0.0068883285857737064, -0.032303888350725174, 0.07302466779947281, -0.035076647996902466, -0.03563516214489937, 0.02606920525431633, 0.05076679587364197, 0.02319413423538208, -0.009693698957562447, -0.047582466155290604, 0.002919740043580532, 0.05556236580014229, -0.013434740714728832, -0.015244543552398682, 0.02350369654595852, -0.002612635027617216, -0.03747796639800072, 0.029851974919438362, -0.017929568886756897, 0.005668936297297478, 0.05689527839422226, 0.08259685337543488, -0.04409698396921158, -0.020461028441786766, -0.05634477734565735, -0.06020928546786308, 0.0912451222538948, 0.04807163402438164, -0.0634351521730423, 0.06615909934043884, 0.0577421560883522, 0.02740497700870037, -0.031116239726543427, 0.04045913368463516, 0.018953057006001472, -0.10272610187530518, -0.08337682485580444, -0.09989166259765625, 0.043805431574583054, 0.02688218280673027, 0.014868419617414474, -0.034112751483917236, -0.027920272201299667, 0.029938897117972374, 0.011456538923084736, -0.06435238569974899, 1.9176078117253987e-33, 0.048523493111133575, -0.016604095697402954, 0.02822207100689411, -0.11147268861532211, 0.08810616284608841, 0.003103473922237754, 0.016116252169013023, 0.01928049698472023, -0.09448369592428207, -0.013867964968085289, -0.027936916798353195, 0.07054978609085083, 0.00670698331668973, 0.01044241338968277, 0.03523870185017586, -0.04192784056067467, -0.018141023814678192, 0.07065747678279877, 0.08403744548559189, 0.061018604785203934, 0.04657676815986633, -0.05936289206147194, -0.009776483289897442, 0.007256131153553724, 0.021935366094112396, 0.012536431662738323, 0.056103043258190155, 0.009438068605959415, 0.04698248580098152, 0.0333077572286129, -0.023015402257442474, -0.08556229621171951, -0.007610197179019451, 0.02038782462477684, 0.03791613131761551, -0.07189298421144485, -0.06980198621749878, -0.0571599155664444, 0.007414001040160656, 0.023433992639183998, -0.00479891849681735, 0.01776411198079586, -0.10016501694917679, -0.04505573958158493, -0.039798036217689514, -0.0009126919903792441, -0.029788795858621597, -0.050578705966472626, 0.03168235346674919, -0.01924174092710018, 0.11106050759553909, 0.017998214811086655, 0.03220384940505028, -0.005235073156654835, 0.06567565351724625, 0.03682543709874153, 0.07076107710599899, -0.022777730599045753, 0.0386214479804039, 0.06058921292424202, -0.07981505990028381, -0.01912962831556797, -0.06236041709780693, 0.035301510244607925, 0.005597411189228296, -0.054670218378305435, 0.047188155353069305, 0.08034088462591171, 0.02310079149901867, -0.004662695806473494, -0.009987777099013329, -0.012309315614402294, 0.06983225792646408, 0.06838769465684891, 0.06794392317533493, -0.011658687144517899, 0.00048192118993029, 0.024348696693778038, -0.027921482920646667, -0.008704553358256817, -0.043475326150655746, -0.02925853431224823, 0.0014699678868055344, 0.047023870050907135, 0.008884437382221222, -0.013704565353691578, -0.02721046283841133, 0.018465938046574593, -0.11433548480272293, 0.03573836013674736, -0.08787364512681961, 0.01614609733223915, 0.07766127586364746, -0.027548586949706078, -0.027749354019761086, -2.5456522191796767e-33, -0.051880232989788055, 0.010609125718474388, -0.034008026123046875, 0.1296122968196869, 0.08679812401533127, -0.05832454934716225, 0.07041805237531662, -0.0008646771311759949, 0.007786416448652744, 0.014124558307230473, -0.05393674597144127, 0.0025924458168447018, 0.0018217518227174878, -0.015095573849976063, 0.035310450941324234, -0.028661319985985756, -0.002545525785535574, -0.13136237859725952, -0.023445023223757744, 0.010562043637037277, -0.004121020436286926, 0.019269153475761414, -0.007250993978232145, 0.004447334911674261, -0.03605847805738449, -0.010661615990102291, -0.05064380168914795, -0.002300841500982642, 0.017919018864631653, -0.01914912648499012, 0.02137143909931183, 0.02259998768568039, 0.006458733230829239, -0.11112105846405029, 0.017941024154424667, -0.04479946568608284, 0.09241434186697006, 0.03140066936612129, 0.03923803195357323, -0.009351697750389576, -0.017988082021474838, -0.009879431687295437, 0.040496826171875, -0.03998282924294472, -0.026738766580820084, 0.09971068054437637, -0.008086304180324078, 0.061276037245988846, -0.032826852053403854, -0.0253130029886961, -0.056416332721710205, 0.015237093903124332, -0.02254786528646946, 0.033029746264219284, 0.025828614830970764, -0.038142841309309006, 0.017419449985027313, 0.002233796985819936, -0.138339564204216, -0.03583831340074539, -0.024587463587522507, -0.03458356112241745, 0.10121329873800278, 0.0017955348594114184, -0.02243809774518013, 0.02294073812663555, 0.01986278034746647, 0.013287126086652279, -0.14369899034500122, -0.008195938542485237, -0.0010208070743829012, -0.1172943189740181, -0.004770664963871241, 0.09437159448862076, 0.008753892965614796, -0.0027984054759144783, -0.015055130235850811, -0.050968363881111145, 0.06796887516975403, 0.08026447147130966, 0.0015055668773129582, 0.08716031163930893, 0.003129352815449238, -0.008337263949215412, 0.14612466096878052, -0.011662853881716728, -0.01087844930589199, -0.015590676106512547, 0.04087084159255028, 0.031123900786042213, -0.08632262796163559, 0.04776468500494957, -0.09908502548933029, -0.0096565131098032, 0.0012567397207021713, -3.492227307333451e-08, -0.03131479024887085, 0.039830658584833145, -0.03799265995621681, -0.0024163657799363136, 0.04210447892546654, 0.013701516203582287, 0.053717710077762604, 0.1696431189775467, 0.041706692427396774, -0.03259803727269173, 0.06020280718803406, -0.08854134380817413, -0.013443593867123127, 0.017301753163337708, 0.06332704424858093, 0.007233734242618084, 0.052556343376636505, -0.0778719037771225, -0.02314665913581848, -0.02860150672495365, -0.007205752190202475, 0.03254684805870056, -0.0012815148802474141, 0.02611367031931877, -0.06734127551317215, -0.011466773226857185, 0.16997094452381134, 0.037703197449445724, -0.02536364272236824, -0.06489993631839752, -0.026916049420833588, 0.008996563032269478, 0.007384839002043009, -0.025420036166906357, 0.07738093286752701, -0.005997790489345789, 0.006048841867595911, 0.03464819863438606, 0.05387331545352936, 0.00037185035762377083, 0.04431060329079628, 0.022498739883303642, 0.015599384903907776, -0.03256010636687279, -0.0007507397094741464, -0.021523743867874146, -0.018510466441512108, 0.0167588759213686, -0.005927710328251123, 0.08524975925683975, 0.025901226326823235, -0.05621309205889702, -0.006224010139703751, 0.09505369514226913, 0.1378629207611084, 0.006168277468532324, 0.02945222333073616, -0.1219346821308136, 0.06274287402629852, 0.03979019075632095, 0.043842218816280365, 0.008064117282629013, 0.04107089713215828, -0.007163587491959333]', 0: 3}, {1: '3_0', 2: 'Google Beam is an innovative video communication platform that builds on the research of Project Starline. It uses AI, 3D imaging, and light field rendering to create immersive, lifelike video calls. Designed to replicate in-person interaction, Beam allows users to see life-sized, three-dimensional representations of each other without the need for headsets. This breakthrough makes remote conversations feel natural—capturing facial expressions, eye contact, and subtle gestures that traditional video conferencing often misses. Beam reduces meeting fatigue and enhances engagement, making it ideal for enterprise collaboration, interviews, and virtual presence scenarios. Powered by Google AI, Beam represents a significant leap in communication technology. Major companies like Salesforce, Deloitte, and NEC are already exploring its impact on digital collaboration. Google is partnering with HP to build and distribute Beam hardware, designed to work with existing productivity and video tools. Currently in limited early access for enterprise partners, Google Beam aims to redefine virtual meetings by bridging the gap between digital and physical presence. It’s a promising step toward more human and effective remote interactions.', 3: 'Google Beam: 3D Communication Powered by AI. Google Beam is an innovative video communication platform that builds on the research of Project Starline. It uses AI, 3D imaging, and light field rendering to create immersive, lifelike video calls. Designed to replicate in-person interaction, Beam allows users to see life-sized, three-dimensional representations of each other without the need for headsets. This breakthrough makes remote conversations feel natural—capturing facial expressions, eye contact, and subtle gestures that traditional video conferencing often misses. Beam reduces meeting fatigue and enhances engagement, making it ideal for enterprise collaboration, interviews, and virtual presence scenarios. Powered by Google AI, Beam represents a significant leap in communication technology. Major companies like Salesforce, Deloitte, and NEC are already exploring its impact on digital collaboration. Google is partnering with HP to build and distribute Beam hardware, designed to work with existing productivity and video tools. Currently in limited early access for enterprise partners, Google Beam aims to redefine virtual meetings by bridging the gap between digital and physical presence. It’s a promising step toward more human and effective remote interactions.', 4: \"{'title': 'Google Beam: 3D Communication Powered by AI', 'keywords': ['Google Beam', 'Project Starline', '3D video', 'AI communication', 'real-time meetings'], 'tags': ['AI', 'Communication', '3D Technology', 'Remote Work', 'Enterprise Tech']}\", 5: '[-0.08768321573734283, -0.10005483776330948, 0.03568415343761444, -0.08212734013795853, -0.01738724298775196, -0.0572899766266346, 0.0060171508230268955, -0.06130814179778099, 0.001428459770977497, -0.08556339889764786, -0.0952070876955986, 0.041995689272880554, -0.1202535480260849, -0.01066769752651453, 0.043444711714982986, -0.04269992187619209, 0.12283974885940552, -0.040432173758745193, -0.007321997079998255, 0.055041611194610596, 0.003923223819583654, -0.06601600348949432, -0.0031789555214345455, -0.015608920715749264, 0.013913601636886597, 0.007402278017252684, 0.014218656346201897, -0.08461038023233414, 0.02698861062526703, -0.04228398576378822, -0.02136833965778351, 0.015536555089056492, 0.04961182549595833, 0.07363967597484589, -0.10466285794973373, 0.005210317671298981, 0.07844141870737076, -0.033175189048051834, -0.1322903037071228, -0.0868099182844162, -0.07534613460302353, -0.04554476961493492, -0.0164048932492733, -0.026506656780838966, 0.007949941791594028, -0.0182399433106184, -0.0517999492585659, -0.07252563536167145, 0.023599285632371902, 0.05206107348203659, -0.057303350418806076, -0.09666356444358826, 0.044495485723018646, 0.004883192013949156, -0.05511088669300079, 0.07372823357582092, -0.006772668566554785, 0.030569585040211678, 0.05252925306558609, 0.11124249547719955, 0.008853021077811718, -0.0019371907692402601, 0.03833448886871338, 0.037561409175395966, -0.015580684877932072, 0.03981165587902069, 0.049070168286561966, -0.006949519272893667, -0.015260118991136551, -0.0832832083106041, -0.06730788946151733, 0.03521919623017311, -0.03924863785505295, -0.02580232359468937, -0.05239849165081978, -0.015273735858500004, -0.0011428530560806394, -0.0840705931186676, 0.06744348257780075, 0.06217053905129433, 0.035872187465429306, -0.0127906808629632, -0.036415282636880875, 0.039919137954711914, -0.03077578917145729, -0.0368623323738575, -0.0354931466281414, 0.08335445821285248, -0.047571875154972076, 0.023561187088489532, -0.12263979017734528, 0.03724728524684906, -0.06913828104734421, -0.04302017018198967, 0.0026693479157984257, -0.03369182348251343, -0.0255685243755579, -0.10069906711578369, -0.013446783646941185, 0.0481596440076828, -0.044240694493055344, 0.025013798847794533, 0.04790237545967102, -0.0463143065571785, -0.06570546329021454, 0.009200280532240868, -0.05090705305337906, 0.04666191712021828, 0.016258712857961655, 0.02338799461722374, -0.01469238381832838, -0.02688952162861824, -0.037063490599393845, -0.04772617295384407, 0.036033328622579575, 0.013595202006399632, -0.07186692208051682, -0.012121873907744884, 0.13698242604732513, -0.054272957146167755, 0.09579470753669739, -0.018000900745391846, 0.03515244647860527, 0.035747576504945755, 0.015358648262917995, -0.020873773843050003, -0.03420866280794144, 2.373642498165836e-33, 0.004087001550942659, 0.09396524727344513, 0.008987114764750004, 0.007892079651355743, 0.02842218615114689, 0.03252958133816719, -0.023905858397483826, 0.05588100850582123, -0.07259924709796906, -0.044402215629816055, 0.06960228830575943, -0.003998559899628162, 0.08624311536550522, 0.036521703004837036, 0.07360218465328217, -0.04819520562887192, 0.007478090934455395, 0.05898915231227875, -0.019743043929338455, -0.0237325020134449, -0.015354865230619907, -0.05497432500123978, 0.005782193969935179, 0.06376569718122482, 0.013976055197417736, 0.0641258955001831, 0.03309528902173042, 0.004411126486957073, 0.0591343455016613, -0.011793381534516811, -0.04374219477176666, 0.01053654495626688, 0.08095346391201019, 0.0012396350502967834, 0.06581103056669235, -0.0025814322289079428, -0.07841470092535019, -0.11471793055534363, -0.02738858386874199, 0.08644800633192062, -0.030907731503248215, 0.057290345430374146, -0.03507879748940468, -0.06745754182338715, -0.03848349303007126, 0.037257324904203415, 0.05201214179396629, 0.011970894411206245, -0.06959062069654465, 0.029294224455952644, -0.009705460630357265, 0.06902290880680084, -0.05948861688375473, 0.005885870195925236, 0.0029501882381737232, -0.03473135828971863, 0.05152532830834389, 0.007587505504488945, 0.056073691695928574, -0.08477016538381577, 0.012235607951879501, 0.023133469745516777, 0.001964848255738616, 0.046193916350603104, -0.028170393779873848, -0.0067735956981778145, -0.019663318991661072, 0.062182217836380005, -0.005953589919954538, 0.022341269999742508, 0.012766437605023384, 0.034293413162231445, 0.011621139012277126, -0.01770191825926304, 0.0020938110537827015, 0.021433182060718536, -0.016592945903539658, 0.03301786258816719, 0.021443868055939674, 0.05710439011454582, -0.049771927297115326, -0.028453068807721138, 0.014465024694800377, -0.007035637740045786, 0.045344434678554535, -0.049593571573495865, -0.04927557706832886, -0.0314156599342823, -0.12164126336574554, 0.04815191403031349, -0.03145309537649155, 0.01949147693812847, 0.02038230560719967, 0.06463321298360825, -0.016771676018834114, -2.316360658450266e-33, 0.007842582650482655, 0.034079741686582565, -0.03585878387093544, -0.0024113233666867018, 0.1221923902630806, -0.008060518652200699, 0.08166041225194931, -0.06345520168542862, 0.025417208671569824, 0.02269379235804081, -0.03757006302475929, 0.0024516331031918526, -0.010479165241122246, 0.03265863656997681, 0.047720059752464294, -0.001924882992170751, 0.060983091592788696, -0.12869161367416382, -0.07256944477558136, 0.008958404883742332, 0.14701144397258759, 0.05159146711230278, 0.006388494744896889, 0.02738032117486, -0.004137595649808645, -0.03915055841207504, 0.051600340753793716, -0.041472822427749634, 0.022494586184620857, 0.004335323814302683, 0.039017267525196075, -0.00763319618999958, -0.0055204397067427635, -0.01936323009431362, 0.05861395597457886, 0.06448338180780411, 0.06707989424467087, -0.003935203887522221, 0.014955636113882065, -0.06120843440294266, 0.1013767346739769, 0.050216082483530045, -0.015347855165600777, -0.034159790724515915, 0.011919219978153706, 0.05818180739879608, -0.08250715583562851, 0.052551671862602234, -0.09590791910886765, -0.02140170894563198, -0.019470779225230217, 0.021486302837729454, -0.08193112909793854, -0.08036662638187408, -0.09226694703102112, -0.07088422775268555, 0.006174936890602112, 0.03408947214484215, -0.0005518404068425298, -0.018194660544395447, 0.02761955000460148, -0.04814797639846802, 0.0399932935833931, -0.023555945605039597, 0.0016058675246313214, 0.036425698548555374, 0.0400175042450428, 0.11064691096544266, -0.0003956275468226522, 0.018695760518312454, 0.050038937479257584, -0.059090059250593185, 0.06259125471115112, 0.0765070840716362, 0.002242499962449074, 0.007463021669536829, 0.023093560710549355, -0.04647943750023842, 0.022281557321548462, -0.028783464804291725, 0.02034471184015274, 0.016215592622756958, 0.08534683287143707, 0.023307310417294502, 0.10140694677829742, 0.10804024338722229, -0.034638408571481705, 0.04119233787059784, -0.08908803015947342, 0.035973694175481796, -0.043563228100538254, 0.0729607418179512, -0.01990627683699131, 0.020396526902914047, -0.018512193113565445, -4.470294001635011e-08, -0.02850918285548687, -0.007024200167506933, -0.032094620168209076, -0.060947760939598083, -0.04950270801782608, -0.058581385761499405, 0.051795151084661484, 0.030354613438248634, 0.051092080771923065, -0.054129865020513535, -0.0013695991365239024, -0.10446909070014954, 0.019490908831357956, 0.08895456045866013, 0.0866464376449585, 0.028892185539007187, -0.044817015528678894, -0.029643280431628227, -0.031061984598636627, 0.03367877006530762, 0.03805943951010704, -0.025970734655857086, -0.0036888024769723415, 0.02999047376215458, -0.04606621712446213, 0.021130019798874855, 0.0333632156252861, 0.048508476465940475, -0.02722831629216671, -0.04446767270565033, -0.05041423439979553, 0.08033314347267151, -0.025748170912265778, -0.014920808374881744, 0.047536179423332214, -0.053025342524051666, -0.021161098033189774, -0.021410807967185974, 0.0544908307492733, 0.023105554282665253, -0.006709648296236992, 0.029420357197523117, -0.02852577157318592, 0.00629047304391861, 0.023637467995285988, 0.040019821375608444, 0.02754388563334942, -0.14894872903823853, -0.006592150777578354, 0.04156772792339325, -0.03889602795243263, -0.0013589700683951378, -0.03000982291996479, 0.0449916310608387, 0.08881128579378128, 0.04516582563519478, 0.13244342803955078, -0.008844813331961632, 0.0781707838177681, 0.03751348704099655, 0.0604221411049366, 0.017812488600611687, -0.029359012842178345, 0.02957269176840782]', 0: 4}])\n",
       "              .draw('full-hold');\n",
       "          });\n",
       "        }\n",
       "            </script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "        if (typeof window.interactive_beam_jquery == 'undefined') {\n",
       "          var jqueryScript = document.createElement('script');\n",
       "          jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
       "          jqueryScript.type = 'text/javascript';\n",
       "          jqueryScript.onload = function() {\n",
       "            var datatableScript = document.createElement('script');\n",
       "            datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
       "            datatableScript.type = 'text/javascript';\n",
       "            datatableScript.onload = function() {\n",
       "              window.interactive_beam_jquery = jQuery.noConflict(true);\n",
       "              window.interactive_beam_jquery(document).ready(function($){\n",
       "                \n",
       "            $(\"#progress_indicator_ef090119901644a31067b90f8d98d385\").remove();\n",
       "              });\n",
       "            }\n",
       "            document.head.appendChild(datatableScript);\n",
       "          };\n",
       "          document.head.appendChild(jqueryScript);\n",
       "        } else {\n",
       "          window.interactive_beam_jquery(document).ready(function($){\n",
       "            \n",
       "            $(\"#progress_indicator_ef090119901644a31067b90f8d98d385\").remove();\n",
       "          });\n",
       "        }"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class DocumentSplitterDoFn(beam.DoFn):\n",
    "    def setup(self):\n",
    "        # The `chunk_size` parameter is constrained by the embedding model we’re using.\n",
    "        # Since we’re using `sentence-transformers/all-MiniLM-L6-v2`, which has a maximum\n",
    "        # token limit of ~384 tokens, we need to ensure chunk sizes stay well within that limit.\n",
    "        # Given that each document in our dataset contains approximately 331 tokens, using a chunk\n",
    "        # size of 256 allows us to preserve nearly the most semantic meaning of each entry while\n",
    "        # staying safely under the model’s token limit.\n",
    "        #\n",
    "        # For simplicity, We'll use sentence splitting as the chunking strategy for simplicity. Ideally,\n",
    "        # we would pass a tokenizer here — preferably the same one used by the retriever to ensure\n",
    "        # consistency. However, in this example, we are not using a tokenizer.\n",
    "        from llama_index.core.text_splitter import SentenceSplitter\n",
    "        chunk_size, chunk_overlap = 256, 20\n",
    "        self.llama_txt_splitter = SentenceSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)\n",
    "\n",
    "    def process(self, element: Dict[str, Any]) -> List[Chunk]:\n",
    "        id_field, content_field = 'id', 'content'\n",
    "        metadata_fields = [\"title\", \"keywords\", \"tags\"]\n",
    "        global_doc_id = element.get('id', str(uuid.uuid4()))\n",
    "        text_content = element.get('content', '')\n",
    "        splits = self.llama_txt_splitter.split_text(text_content)\n",
    "        for i, split in enumerate(splits):\n",
    "            local_doc_id = f\"{global_doc_id}_{i}\"\n",
    "            yield Chunk(id=local_doc_id, content=Content(split), metadata={f:element[f] for f in metadata_fields})\n",
    "\n",
    "class ChunkingTransformProvider(ChunkingTransformProvider):\n",
    "    def get_splitter_transform(self) -> beam.PTransform[beam.PCollection[Dict[str, Any]], beam.PCollection[Chunk]]:\n",
    "        return beam.ParDo(DocumentSplitterDoFn())\n",
    "\n",
    "class IndexToVectorDBDoFn(beam.DoFn):\n",
    "    def __init__(self, collection_name: str, batch_size: int = 100):\n",
    "        self.collection_name = collection_name\n",
    "        self.batch_size = batch_size\n",
    "\n",
    "    def setup(self):\n",
    "        self._client = MilvusClient(**milvus_connection_parameters.__dict__)\n",
    "\n",
    "    def start_bundle(self):\n",
    "        self._batch = []\n",
    "\n",
    "    def process(self, doc: Chunk):\n",
    "        doc_to_index = {\n",
    "            \"id\": doc.id,\n",
    "            \"content\": doc.content.text,\n",
    "            \"title_and_content\": f\"{doc.metadata['title']}. {doc.content.text}\",\n",
    "            \"metadata\": doc.metadata,\n",
    "            \"embedding\": doc.embedding.dense_embedding,\n",
    "        }\n",
    "        self._batch.append(doc_to_index)\n",
    "\n",
    "        if len(self._batch) >= self.batch_size:\n",
    "            self._flush_batch()\n",
    "\n",
    "        yield doc_to_index\n",
    "\n",
    "    def finish_bundle(self):\n",
    "        if self._batch:\n",
    "            self._flush_batch()\n",
    "\n",
    "    def _flush_batch(self):\n",
    "        if self._batch:\n",
    "            # Upsert API gives us a built-in idempotency over the insert API.\n",
    "            result = self._client.upsert(collection_name=self.collection_name, data=self._batch)\n",
    "            print(f\"Upserted batch of {len(self._batch)} documents. Result: {result}\")\n",
    "            self._batch = []\n",
    "\n",
    "huggingface_embedder = HuggingfaceTextEmbeddings(\n",
    "    model_name=embedding_model_config[\"name\"],\n",
    "    max_seq_length=embedding_model_config[\"token_limit\"])\n",
    "\n",
    "with beam.Pipeline() as pipeline:\n",
    "    data_transformed = (\n",
    "        pipeline\n",
    "        | 'Creating Documents' >> beam.Create(corpus)\n",
    "        | 'Converting to Chunks' >> MLTransform(\n",
    "            write_artifact_location=tempfile.mkdtemp()).with_transform(ChunkingTransformProvider())\n",
    "        | 'Generating Embeddings' >> MLTransform(\n",
    "            write_artifact_location=tempfile.mkdtemp()).with_transform(huggingface_embedder)\n",
    "        | 'Indexing to Vector DB' >> beam.ParDo(IndexToVectorDBDoFn(collection_name=collection_name))\n",
    "    )\n",
    "\n",
    "ib.show(data_transformed)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea478136-2ca8-4fee-bb1e-6bfcc2e97c93",
   "metadata": {},
   "source": [
    "## Milvus Beam Enrichment Handler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9ad2509-3e5d-42e8-b565-ecccde38b8f4",
   "metadata": {},
   "source": [
    "### Prep for Milvus Beam Enrichment Handler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "4911e8cc-10f1-4d21-9251-1b756b61f2c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "class FormatAndPrintResults(beam.PTransform):\n",
    "    def expand(self, pcoll):\n",
    "        return pcoll | beam.Map(self.format_and_print)\n",
    "    \n",
    "    @staticmethod\n",
    "    def format_and_print(chunk):\n",
    "        # Create a clean structure to display.\n",
    "        formatted_result = {\n",
    "            \"query\": chunk.content.text,\n",
    "            \"query_embedding\": FormatAndPrintResults.get_embedding_count(chunk),\n",
    "            \"results\": []\n",
    "        }\n",
    "        \n",
    "        # Extract the enrichment data\n",
    "        enrichment_data = chunk.metadata.get('enrichment_data', defaultdict(list))\n",
    "        \n",
    "        # Format each result with its distance score\n",
    "        for i in range(len(enrichment_data.get('id', []))):\n",
    "            result = {\n",
    "                \"id\": enrichment_data['id'][i],\n",
    "                \"distance\": round(enrichment_data['distance'][i], 4),\n",
    "                \"fields\": enrichment_data['fields'][i] if i < len(enrichment_data.get('fields', [])) else {}\n",
    "            }\n",
    "            formatted_result[\"results\"].append(result)\n",
    "        \n",
    "        # Sort by distance in descending order (highest/best first)\n",
    "        formatted_result[\"results\"] = sorted(formatted_result[\"results\"], key=lambda x: x[\"distance\"], reverse=True)\n",
    "\n",
    "        # Print the formatted JSON\n",
    "        print_json(data=formatted_result)\n",
    "        \n",
    "        # Return the original chunk for further processing if needed\n",
    "        return chunk\n",
    "\n",
    "    @staticmethod\n",
    "    def get_embedding_count(chunk):\n",
    "        if chunk.embedding:\n",
    "            if chunk.embedding.dense_embedding:\n",
    "                return len(chunk.embedding.dense_embedding)\n",
    "            if chunk.embedding.sparse_embedding:\n",
    "                return len(chunk.embedding.sparse_embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "656110c9-1360-49fd-ba17-f55f2257f127",
   "metadata": {},
   "source": [
    "### Vector Search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d165518-b27b-40a8-ae0a-42342df3c1eb",
   "metadata": {},
   "source": [
    "Let’s choose a deliberate query that illustrates the unique benefits of pure vector search, especially its ability to grasp semantic meaning:\n",
    "\n",
    "Query: `How do I process large datasets efficiently?`\n",
    "\n",
    "This query demonstrates vector search advantages because:\n",
    "\n",
    "- **Dense vector (semantic) contribution:** The semantic component understands the conceptual intent of \"processing large datasets efficiently,\" connecting it to frameworks like **Apache Beam** and **Google Cloud Dataflow**, even if those terms aren't in the query.\n",
    "- **Overcoming keyword limitations:** For conversational queries like this, traditional keyword search struggles. Vector search moves beyond exact lexical matching to find documents that semantically answer the \"how-to\" aspect.\n",
    "- **Vector search advantage:** Documents describing solutions like **Apache Beam** (e.g., Document #1) rank highest. Vector search understands that Beam's \"unified programming model for defining and executing data processing pipelines\" directly addresses the query's need for efficient large dataset processing, even without an exact phrase match, by prioritizing based on deep semantic alignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "74db1238-0a04-4e08-818d-5bce8f09006b",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"How do I process large datasets efficiently?\"\n",
    "query_chunk = Chunk(content=Content(text=query))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "79e16531-8bec-4b4b-9ed3-cebd705480e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "search_parameters = MilvusSearchParameters(\n",
    "    collection_name=collection_name,\n",
    "    search_strategy=VectorSearchParameters(limit=10, anns_field=\"embedding\"),\n",
    "    output_fields=[\"metadata\",\"content\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "cbef1911-6464-4ba1-8974-ed00896c7e8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "collection_load_parameters = MilvusCollectionLoadParameters() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "f0481286-3f2b-4690-a2f6-a5a00de3ff34",
   "metadata": {},
   "outputs": [],
   "source": [
    "milvus_handler = MilvusSearchEnrichmentHandler(\n",
    "    connection_parameters=milvus_connection_parameters,\n",
    "    search_parameters=search_parameters,\n",
    "    collection_load_parameters=collection_load_parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "35ee37f2-60cd-4d5d-aef6-aed4fda79161",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Union[Tuple[apache_beam.pvalue.PCollection[~MLTransformOutputT], apache_beam.pvalue.PCollection[apache_beam.pvalue.Row]], apache_beam.pvalue.PCollection[~MLTransformOutputT]] instead.\n",
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Union[Tuple[apache_beam.pvalue.PCollection[~MLTransformOutputT], apache_beam.pvalue.PCollection[apache_beam.pvalue.Row]], apache_beam.pvalue.PCollection[~MLTransformOutputT]] instead.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"query\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"How do I process large datasets efficiently?\"</span>,\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"query_embedding\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span>,\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"results\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "    <span style=\"font-weight: bold\">{</span>\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"id\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"1_0\"</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"distance\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.3657</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"fields\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"content\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam is an open-source framework that provides a consistent programming model for both batch and streaming data processing. Developed originally by Google, it allows developers to write pipelines that can run on multiple engines, such as Apache Flink, Spark, and Google Cloud Dataflow. Beam uses abstractions like PCollections (data containers) and PTransforms (operations) to define the flow of data. The framework promotes portability through its runner architecture, letting the same pipeline execute on different backends. Support for multiple SDKs, including Java and Python, makes it accessible for a broad audience. Key features include support for event time, windowing, triggers, and stateful processing, which are essential for handling real-time data effectively. Beam is ideal for building ETL jobs, real-time analytics, and machine learning data pipelines. It helps teams focus on logic rather than infrastructure, offering flexibility and scalability in handling unbounded and bounded data sources. Apache Beam also supports a wide range of connectors for both input and output, including Kafka, BigQuery, and JDBC-based systems. This makes it easy to integrate Beam into existing data ecosystems. Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing.\"</span>,\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"metadata\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"title\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam: Unified Model for Batch and Streaming Data\"</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"keywords\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"stream processing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"batch processing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"data pipelines\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"SDK\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"tags\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Data Engineering\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Open Source\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Streaming\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Batch\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Big Data\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "      <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>,\n",
       "    <span style=\"font-weight: bold\">{</span>\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"id\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"2_1\"</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"distance\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.3369</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"fields\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"content\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"For developers, Dataflow provides local testing capabilities and a unified logging system through Cloud Logging. It also supports SQL-based pipeline definitions using BigQuery, which lowers the barrier to entry for analysts and data engineers. Dataflow’s streaming engine significantly improves performance and reduces costs by decoupling compute and state management. In summary, Google Cloud Dataflow not only simplifies the deployment of Apache Beam pipelines but also enhances them with cloud-native features. Its managed runtime, high availability, and integration with the broader Google Cloud ecosystem make it a powerful tool for modern data processing.\"</span>,\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"metadata\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"title\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud Dataflow: Run Apache Beam in the Cloud\"</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"keywords\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Dataflow\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"serverless\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"stream and batch\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"tags\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Cloud Computing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Data Pipelines\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Serverless\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Enterprise\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "      <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>,\n",
       "    <span style=\"font-weight: bold\">{</span>\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"id\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"2_0\"</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"distance\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.2918</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"fields\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"content\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud Dataflow is a fully managed service that runs Apache Beam pipelines in the cloud. It abstracts away infrastructure management and handles dynamic scaling, load balancing, and fault tolerance. Developers can focus on writing data logic using the Beam SDK and deploy it easily to Google Cloud. Dataflow supports both batch and stream processing and integrates seamlessly with other Google services like BigQuery, Pub/Sub, and Cloud Storage. Its autoscaling capabilities allow it to adapt to changing data volumes, optimizing for cost and performance. Features like monitoring dashboards, job templates, and built-in logging make it suitable for both development and production use. With support for event time processing, stateful functions, and windowing, Dataflow is well-suited for real-time analytics and data transformation tasks. It’s a key component for architects building scalable, cloud-native data platforms. Dataflow also offers templates for common ETL tasks, helping teams get started quickly with minimal setup. Its integration with Cloud Functions and Cloud Composer enables event-driven and orchestrated workflows. Security and compliance are built-in with IAM roles, encryption at rest and in transit, and audit logging, making it suitable for enterprise environments.\"</span>,\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"metadata\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"title\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud Dataflow: Run Apache Beam in the Cloud\"</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"keywords\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Dataflow\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"serverless\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"stream and batch\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"tags\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Cloud Computing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Data Pipelines\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Serverless\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Enterprise\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "      <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>,\n",
       "    <span style=\"font-weight: bold\">{</span>\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"id\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"1_1\"</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"distance\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.2638</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"fields\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"content\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing. The concept of runners enables developers to write once and run anywhere, which is particularly appealing for organizations that want to avoid vendor lock-in. The Beam model is based on a unified programming model that decouples pipeline logic from execution. This makes it easier to reason about time and state in both batch and streaming pipelines. Advanced features like late data handling, watermarks, and session windowing allow for more accurate and meaningful processing of real-world data. Beam also integrates with orchestration tools and monitoring systems, allowing for production-grade deployments. Community support and contributions have grown significantly, making Beam a stable and evolving ecosystem. Many cloud providers offer native support for Beam pipelines, and it's increasingly a core component in modern data platform architectures.\"</span>,\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"metadata\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"title\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam: Unified Model for Batch and Streaming Data\"</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"keywords\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"stream processing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"batch processing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"data pipelines\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"SDK\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"tags\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Data Engineering\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Open Source\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Streaming\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Batch\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Big Data\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "      <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>,\n",
       "    <span style=\"font-weight: bold\">{</span>\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"id\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"3_0\"</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"distance\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.031</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"fields\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"content\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Beam is an innovative video communication platform that builds on the research of Project Starline. It uses AI, 3D imaging, and light field rendering to create immersive, lifelike video calls. Designed to replicate in-person interaction, Beam allows users to see life-sized, three-dimensional representations of each other without the need for headsets. This breakthrough makes remote conversations feel natural—capturing facial expressions, eye contact, and subtle gestures that traditional video conferencing often misses. Beam reduces meeting fatigue and enhances engagement, making it ideal for enterprise collaboration, interviews, and virtual presence scenarios. Powered by Google AI, Beam represents a significant leap in communication technology. Major companies like Salesforce, Deloitte, and NEC are already exploring its impact on digital collaboration. Google is partnering with HP to build and distribute Beam hardware, designed to work with existing productivity and video tools. Currently in limited early access for enterprise partners, Google Beam aims to redefine virtual meetings by bridging the gap between digital and physical presence. It’s a promising step toward more human and effective remote interactions.\"</span>,\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"metadata\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"title\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Beam: 3D Communication Powered by AI\"</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"keywords\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Beam\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Project Starline\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"3D video\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"AI communication\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"real-time meetings\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"tags\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"AI\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Communication\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"3D Technology\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Remote Work\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Enterprise Tech\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "      <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>\n",
       "  <span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"query\"\u001b[0m: \u001b[32m\"How do I process large datasets efficiently?\"\u001b[0m,\n",
       "  \u001b[1;34m\"query_embedding\"\u001b[0m: \u001b[1;36m384\u001b[0m,\n",
       "  \u001b[1;34m\"results\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "    \u001b[1m{\u001b[0m\n",
       "      \u001b[1;34m\"id\"\u001b[0m: \u001b[32m\"1_0\"\u001b[0m,\n",
       "      \u001b[1;34m\"distance\"\u001b[0m: \u001b[1;36m0.3657\u001b[0m,\n",
       "      \u001b[1;34m\"fields\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[1;34m\"content\"\u001b[0m: \u001b[32m\"Apache Beam is an open-source framework that provides a consistent programming model for both batch and streaming data processing. Developed originally by Google, it allows developers to write pipelines that can run on multiple engines, such as Apache Flink, Spark, and Google Cloud Dataflow. Beam uses abstractions like PCollections (data containers) and PTransforms (operations) to define the flow of data. The framework promotes portability through its runner architecture, letting the same pipeline execute on different backends. Support for multiple SDKs, including Java and Python, makes it accessible for a broad audience. Key features include support for event time, windowing, triggers, and stateful processing, which are essential for handling real-time data effectively. Beam is ideal for building ETL jobs, real-time analytics, and machine learning data pipelines. It helps teams focus on logic rather than infrastructure, offering flexibility and scalability in handling unbounded and bounded data sources. Apache Beam also supports a wide range of connectors for both input and output, including Kafka, BigQuery, and JDBC-based systems. This makes it easy to integrate Beam into existing data ecosystems. Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing.\"\u001b[0m,\n",
       "        \u001b[1;34m\"metadata\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "          \u001b[1;34m\"title\"\u001b[0m: \u001b[32m\"Apache Beam: Unified Model for Batch and Streaming Data\"\u001b[0m,\n",
       "          \u001b[1;34m\"keywords\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Apache Beam\"\u001b[0m,\n",
       "            \u001b[32m\"stream processing\"\u001b[0m,\n",
       "            \u001b[32m\"batch processing\"\u001b[0m,\n",
       "            \u001b[32m\"data pipelines\"\u001b[0m,\n",
       "            \u001b[32m\"SDK\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m,\n",
       "          \u001b[1;34m\"tags\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Data Engineering\"\u001b[0m,\n",
       "            \u001b[32m\"Open Source\"\u001b[0m,\n",
       "            \u001b[32m\"Streaming\"\u001b[0m,\n",
       "            \u001b[32m\"Batch\"\u001b[0m,\n",
       "            \u001b[32m\"Big Data\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "      \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m,\n",
       "    \u001b[1m{\u001b[0m\n",
       "      \u001b[1;34m\"id\"\u001b[0m: \u001b[32m\"2_1\"\u001b[0m,\n",
       "      \u001b[1;34m\"distance\"\u001b[0m: \u001b[1;36m0.3369\u001b[0m,\n",
       "      \u001b[1;34m\"fields\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[1;34m\"content\"\u001b[0m: \u001b[32m\"For developers, Dataflow provides local testing capabilities and a unified logging system through Cloud Logging. It also supports SQL-based pipeline definitions using BigQuery, which lowers the barrier to entry for analysts and data engineers. Dataflow’s streaming engine significantly improves performance and reduces costs by decoupling compute and state management. In summary, Google Cloud Dataflow not only simplifies the deployment of Apache Beam pipelines but also enhances them with cloud-native features. Its managed runtime, high availability, and integration with the broader Google Cloud ecosystem make it a powerful tool for modern data processing.\"\u001b[0m,\n",
       "        \u001b[1;34m\"metadata\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "          \u001b[1;34m\"title\"\u001b[0m: \u001b[32m\"Google Cloud Dataflow: Run Apache Beam in the Cloud\"\u001b[0m,\n",
       "          \u001b[1;34m\"keywords\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Google Cloud\"\u001b[0m,\n",
       "            \u001b[32m\"Dataflow\"\u001b[0m,\n",
       "            \u001b[32m\"Apache Beam\"\u001b[0m,\n",
       "            \u001b[32m\"serverless\"\u001b[0m,\n",
       "            \u001b[32m\"stream and batch\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m,\n",
       "          \u001b[1;34m\"tags\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Cloud Computing\"\u001b[0m,\n",
       "            \u001b[32m\"Data Pipelines\"\u001b[0m,\n",
       "            \u001b[32m\"Google Cloud\"\u001b[0m,\n",
       "            \u001b[32m\"Serverless\"\u001b[0m,\n",
       "            \u001b[32m\"Enterprise\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "      \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m,\n",
       "    \u001b[1m{\u001b[0m\n",
       "      \u001b[1;34m\"id\"\u001b[0m: \u001b[32m\"2_0\"\u001b[0m,\n",
       "      \u001b[1;34m\"distance\"\u001b[0m: \u001b[1;36m0.2918\u001b[0m,\n",
       "      \u001b[1;34m\"fields\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[1;34m\"content\"\u001b[0m: \u001b[32m\"Google Cloud Dataflow is a fully managed service that runs Apache Beam pipelines in the cloud. It abstracts away infrastructure management and handles dynamic scaling, load balancing, and fault tolerance. Developers can focus on writing data logic using the Beam SDK and deploy it easily to Google Cloud. Dataflow supports both batch and stream processing and integrates seamlessly with other Google services like BigQuery, Pub/Sub, and Cloud Storage. Its autoscaling capabilities allow it to adapt to changing data volumes, optimizing for cost and performance. Features like monitoring dashboards, job templates, and built-in logging make it suitable for both development and production use. With support for event time processing, stateful functions, and windowing, Dataflow is well-suited for real-time analytics and data transformation tasks. It’s a key component for architects building scalable, cloud-native data platforms. Dataflow also offers templates for common ETL tasks, helping teams get started quickly with minimal setup. Its integration with Cloud Functions and Cloud Composer enables event-driven and orchestrated workflows. Security and compliance are built-in with IAM roles, encryption at rest and in transit, and audit logging, making it suitable for enterprise environments.\"\u001b[0m,\n",
       "        \u001b[1;34m\"metadata\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "          \u001b[1;34m\"title\"\u001b[0m: \u001b[32m\"Google Cloud Dataflow: Run Apache Beam in the Cloud\"\u001b[0m,\n",
       "          \u001b[1;34m\"keywords\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Google Cloud\"\u001b[0m,\n",
       "            \u001b[32m\"Dataflow\"\u001b[0m,\n",
       "            \u001b[32m\"Apache Beam\"\u001b[0m,\n",
       "            \u001b[32m\"serverless\"\u001b[0m,\n",
       "            \u001b[32m\"stream and batch\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m,\n",
       "          \u001b[1;34m\"tags\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Cloud Computing\"\u001b[0m,\n",
       "            \u001b[32m\"Data Pipelines\"\u001b[0m,\n",
       "            \u001b[32m\"Google Cloud\"\u001b[0m,\n",
       "            \u001b[32m\"Serverless\"\u001b[0m,\n",
       "            \u001b[32m\"Enterprise\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "      \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m,\n",
       "    \u001b[1m{\u001b[0m\n",
       "      \u001b[1;34m\"id\"\u001b[0m: \u001b[32m\"1_1\"\u001b[0m,\n",
       "      \u001b[1;34m\"distance\"\u001b[0m: \u001b[1;36m0.2638\u001b[0m,\n",
       "      \u001b[1;34m\"fields\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[1;34m\"content\"\u001b[0m: \u001b[32m\"Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing. The concept of runners enables developers to write once and run anywhere, which is particularly appealing for organizations that want to avoid vendor lock-in. The Beam model is based on a unified programming model that decouples pipeline logic from execution. This makes it easier to reason about time and state in both batch and streaming pipelines. Advanced features like late data handling, watermarks, and session windowing allow for more accurate and meaningful processing of real-world data. Beam also integrates with orchestration tools and monitoring systems, allowing for production-grade deployments. Community support and contributions have grown significantly, making Beam a stable and evolving ecosystem. Many cloud providers offer native support for Beam pipelines, and it's increasingly a core component in modern data platform architectures.\"\u001b[0m,\n",
       "        \u001b[1;34m\"metadata\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "          \u001b[1;34m\"title\"\u001b[0m: \u001b[32m\"Apache Beam: Unified Model for Batch and Streaming Data\"\u001b[0m,\n",
       "          \u001b[1;34m\"keywords\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Apache Beam\"\u001b[0m,\n",
       "            \u001b[32m\"stream processing\"\u001b[0m,\n",
       "            \u001b[32m\"batch processing\"\u001b[0m,\n",
       "            \u001b[32m\"data pipelines\"\u001b[0m,\n",
       "            \u001b[32m\"SDK\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m,\n",
       "          \u001b[1;34m\"tags\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Data Engineering\"\u001b[0m,\n",
       "            \u001b[32m\"Open Source\"\u001b[0m,\n",
       "            \u001b[32m\"Streaming\"\u001b[0m,\n",
       "            \u001b[32m\"Batch\"\u001b[0m,\n",
       "            \u001b[32m\"Big Data\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "      \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m,\n",
       "    \u001b[1m{\u001b[0m\n",
       "      \u001b[1;34m\"id\"\u001b[0m: \u001b[32m\"3_0\"\u001b[0m,\n",
       "      \u001b[1;34m\"distance\"\u001b[0m: \u001b[1;36m0.031\u001b[0m,\n",
       "      \u001b[1;34m\"fields\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[1;34m\"content\"\u001b[0m: \u001b[32m\"Google Beam is an innovative video communication platform that builds on the research of Project Starline. It uses AI, 3D imaging, and light field rendering to create immersive, lifelike video calls. Designed to replicate in-person interaction, Beam allows users to see life-sized, three-dimensional representations of each other without the need for headsets. This breakthrough makes remote conversations feel natural—capturing facial expressions, eye contact, and subtle gestures that traditional video conferencing often misses. Beam reduces meeting fatigue and enhances engagement, making it ideal for enterprise collaboration, interviews, and virtual presence scenarios. Powered by Google AI, Beam represents a significant leap in communication technology. Major companies like Salesforce, Deloitte, and NEC are already exploring its impact on digital collaboration. Google is partnering with HP to build and distribute Beam hardware, designed to work with existing productivity and video tools. Currently in limited early access for enterprise partners, Google Beam aims to redefine virtual meetings by bridging the gap between digital and physical presence. It’s a promising step toward more human and effective remote interactions.\"\u001b[0m,\n",
       "        \u001b[1;34m\"metadata\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "          \u001b[1;34m\"title\"\u001b[0m: \u001b[32m\"Google Beam: 3D Communication Powered by AI\"\u001b[0m,\n",
       "          \u001b[1;34m\"keywords\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Google Beam\"\u001b[0m,\n",
       "            \u001b[32m\"Project Starline\"\u001b[0m,\n",
       "            \u001b[32m\"3D video\"\u001b[0m,\n",
       "            \u001b[32m\"AI communication\"\u001b[0m,\n",
       "            \u001b[32m\"real-time meetings\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m,\n",
       "          \u001b[1;34m\"tags\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"AI\"\u001b[0m,\n",
       "            \u001b[32m\"Communication\"\u001b[0m,\n",
       "            \u001b[32m\"3D Technology\"\u001b[0m,\n",
       "            \u001b[32m\"Remote Work\"\u001b[0m,\n",
       "            \u001b[32m\"Enterprise Tech\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "      \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m\n",
       "  \u001b[1m]\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with beam.Pipeline() as p:\n",
    "    _ = (\n",
    "        p\n",
    "        | \"Creating Queries\" >> beam.Create([query_chunk])\n",
    "        | 'Generating Embeddings' >> MLTransform(\n",
    "            write_artifact_location=tempfile.mkdtemp()).with_transform(huggingface_embedder)\n",
    "        | \"Enriching W/ Milvus Vector Search\" >> Enrichment(milvus_handler)\n",
    "        | \"Formatting and Printing Results\" >> FormatAndPrintResults())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb626be4-1c1c-4426-a6be-9cc8e385f2c8",
   "metadata": {},
   "source": [
    "### Keyword Search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b30b29dc-0a59-4cff-b8a3-ace6e801b4da",
   "metadata": {},
   "source": [
    "Let’s choose a deliberate query that illustrates the unique benefits of pure keyword search, especially its ability to pinpoint exact textual matches:\n",
    "\n",
    "Query: `Project Starline`\n",
    "\n",
    "This query demonstrates keyword search advantages because:\n",
    "\n",
    "- **Keyword (lexical) contribution:** The query, `Project Starline`, is an exact phrase. Keyword search is designed to prioritize and precisely match such literal strings, acting as an exact textual filter for specific product names or unique identifiers.\n",
    "- **Overcoming vector limitations:** For a highly specific, proper noun like \"Project Starline\", pure vector search might struggle. It could semantically relate to other \"projects\" or \"communication technologies,\" potentially diluting the precision by not inherently prioritizing the exact string match over broader semantic similarity.\n",
    "- **Keyword search advantage:** Only Document 3 (\"Google Beam: 3D Communication Powered by AI\") contains the exact phrase: `Google Beam is an innovative video communication platform that builds on the research of Project Starline.` A keyword search for \"Project Starline\" will exclusively and precisely retrieve Document 3, showcasing its unparalleled accuracy for factual lookups and named entities where the exact string is paramount.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "f159ad87-5153-48bb-87b3-3845d3c76420",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Project Starline\"\n",
    "query_chunk = Chunk(content=Content(text=query))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "8b8cad3e-8a18-464b-8de6-aa4515a653c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "search_parameters = MilvusSearchParameters(\n",
    "    collection_name=collection_name,\n",
    "    search_strategy=KeywordSearchParameters(limit=10,anns_field=\"sparse_embedding\"),\n",
    "    output_fields=[\"metadata\",\"content\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "47cfc650-0b34-4333-9321-19be2e8fdc85",
   "metadata": {},
   "outputs": [],
   "source": [
    "collection_load_parameters = MilvusCollectionLoadParameters()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "4754763b-66bf-4f90-9920-28cef223b536",
   "metadata": {},
   "outputs": [],
   "source": [
    "milvus_handler = MilvusSearchEnrichmentHandler(\n",
    "    connection_parameters=milvus_connection_parameters,\n",
    "    search_parameters=search_parameters,\n",
    "    collection_load_parameters=collection_load_parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "a3db4837-01c7-42d7-b4e8-58d8d361fe93",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Union[Tuple[apache_beam.pvalue.PCollection[~MLTransformOutputT], apache_beam.pvalue.PCollection[apache_beam.pvalue.Row]], apache_beam.pvalue.PCollection[~MLTransformOutputT]] instead.\n",
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Union[Tuple[apache_beam.pvalue.PCollection[~MLTransformOutputT], apache_beam.pvalue.PCollection[apache_beam.pvalue.Row]], apache_beam.pvalue.PCollection[~MLTransformOutputT]] instead.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"query\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Project Starline\"</span>,\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"query_embedding\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span>,\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"results\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "    <span style=\"font-weight: bold\">{</span>\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"id\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"3_0\"</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"distance\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.8536</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"fields\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"content\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Beam is an innovative video communication platform that builds on the research of Project Starline. It uses AI, 3D imaging, and light field rendering to create immersive, lifelike video calls. Designed to replicate in-person interaction, Beam allows users to see life-sized, three-dimensional representations of each other without the need for headsets. This breakthrough makes remote conversations feel natural—capturing facial expressions, eye contact, and subtle gestures that traditional video conferencing often misses. Beam reduces meeting fatigue and enhances engagement, making it ideal for enterprise collaboration, interviews, and virtual presence scenarios. Powered by Google AI, Beam represents a significant leap in communication technology. Major companies like Salesforce, Deloitte, and NEC are already exploring its impact on digital collaboration. Google is partnering with HP to build and distribute Beam hardware, designed to work with existing productivity and video tools. Currently in limited early access for enterprise partners, Google Beam aims to redefine virtual meetings by bridging the gap between digital and physical presence. It’s a promising step toward more human and effective remote interactions.\"</span>,\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"metadata\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"title\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Beam: 3D Communication Powered by AI\"</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"keywords\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Beam\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Project Starline\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"3D video\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"AI communication\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"real-time meetings\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"tags\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"AI\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Communication\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"3D Technology\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Remote Work\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Enterprise Tech\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "      <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>\n",
       "  <span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"query\"\u001b[0m: \u001b[32m\"Project Starline\"\u001b[0m,\n",
       "  \u001b[1;34m\"query_embedding\"\u001b[0m: \u001b[1;36m384\u001b[0m,\n",
       "  \u001b[1;34m\"results\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "    \u001b[1m{\u001b[0m\n",
       "      \u001b[1;34m\"id\"\u001b[0m: \u001b[32m\"3_0\"\u001b[0m,\n",
       "      \u001b[1;34m\"distance\"\u001b[0m: \u001b[1;36m2.8536\u001b[0m,\n",
       "      \u001b[1;34m\"fields\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[1;34m\"content\"\u001b[0m: \u001b[32m\"Google Beam is an innovative video communication platform that builds on the research of Project Starline. It uses AI, 3D imaging, and light field rendering to create immersive, lifelike video calls. Designed to replicate in-person interaction, Beam allows users to see life-sized, three-dimensional representations of each other without the need for headsets. This breakthrough makes remote conversations feel natural—capturing facial expressions, eye contact, and subtle gestures that traditional video conferencing often misses. Beam reduces meeting fatigue and enhances engagement, making it ideal for enterprise collaboration, interviews, and virtual presence scenarios. Powered by Google AI, Beam represents a significant leap in communication technology. Major companies like Salesforce, Deloitte, and NEC are already exploring its impact on digital collaboration. Google is partnering with HP to build and distribute Beam hardware, designed to work with existing productivity and video tools. Currently in limited early access for enterprise partners, Google Beam aims to redefine virtual meetings by bridging the gap between digital and physical presence. It’s a promising step toward more human and effective remote interactions.\"\u001b[0m,\n",
       "        \u001b[1;34m\"metadata\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "          \u001b[1;34m\"title\"\u001b[0m: \u001b[32m\"Google Beam: 3D Communication Powered by AI\"\u001b[0m,\n",
       "          \u001b[1;34m\"keywords\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Google Beam\"\u001b[0m,\n",
       "            \u001b[32m\"Project Starline\"\u001b[0m,\n",
       "            \u001b[32m\"3D video\"\u001b[0m,\n",
       "            \u001b[32m\"AI communication\"\u001b[0m,\n",
       "            \u001b[32m\"real-time meetings\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m,\n",
       "          \u001b[1;34m\"tags\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"AI\"\u001b[0m,\n",
       "            \u001b[32m\"Communication\"\u001b[0m,\n",
       "            \u001b[32m\"3D Technology\"\u001b[0m,\n",
       "            \u001b[32m\"Remote Work\"\u001b[0m,\n",
       "            \u001b[32m\"Enterprise Tech\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "      \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m\n",
       "  \u001b[1m]\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with beam.Pipeline() as p:\n",
    "    _ = (\n",
    "        p\n",
    "        | \"Creating Queries\" >> beam.Create([query_chunk])\n",
    "        | 'Generating Embeddings' >> MLTransform(\n",
    "            write_artifact_location=tempfile.mkdtemp()).with_transform(huggingface_embedder)\n",
    "        | \"Enriching W/ Milvus Keyword Search\" >> Enrichment(milvus_handler)\n",
    "        | \"Formatting and Printing Results\" >> FormatAndPrintResults()\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de344931-4f2e-473d-bd53-c2708c1d1bcc",
   "metadata": {},
   "source": [
    "### Hybrid Search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e65b2158-5dce-46d1-80de-3c8047419224",
   "metadata": {},
   "source": [
    "Let’s choose a deliberate query that illustrates the unique benefits of hybrid search:\n",
    "\n",
    "Query: `real-time data processing systems`\n",
    "\n",
    "This query demonstrates hybrid search advantages because:\n",
    "\n",
    "* **Dense vector (semantic) contribution:** Will understand the conceptual relationship between \"real-time processing\" and \"streaming\" (found in docs #1 and #2)\n",
    "* **Sparse vector (keyword) contribution:** Will match exact terms like \"data\" and \"processing\" (found in docs #1 and #2)\n",
    "* **Hybrid advantage:** Document #1 about Apache Beam should rank highest since it contains more specific technical details about real-time processing capabilities like \"event time,\" \"triggers,\" and \"stateful processing\" - even though the exact phrase \"real-time data processing\" doesn't appear in any document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "172b6c80-2a03-49d0-afc7-12bb0a4dc989",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"real-time data processing system\"\n",
    "query_chunk = Chunk(content=Content(text=query))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "eb6d951c-0def-45cc-84a4-b6f7b7575f23",
   "metadata": {},
   "outputs": [],
   "source": [
    "hybrid_search_parameters = HybridSearchParameters(\n",
    "    vector=VectorSearchParameters(limit=10,anns_field=\"embedding\"),\n",
    "    keyword=KeywordSearchParameters(limit=10,anns_field=\"sparse_embedding\"),\n",
    "    ranker=RRFRanker(3),\n",
    "    limit=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "b339c498-d229-42e6-b439-b29eb107b533",
   "metadata": {},
   "outputs": [],
   "source": [
    "search_parameters = MilvusSearchParameters(\n",
    "    collection_name=collection_name,\n",
    "    search_strategy=hybrid_search_parameters,\n",
    "    output_fields=[\"metadata\", \"content\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "b346abe6-03c9-4b28-a0fb-74936b9f3a06",
   "metadata": {},
   "outputs": [],
   "source": [
    "collection_load_parameters = MilvusCollectionLoadParameters() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "ab27810d-40a8-4b6a-bc82-441e13763ebc",
   "metadata": {},
   "outputs": [],
   "source": [
    "milvus_handler = MilvusSearchEnrichmentHandler(\n",
    "    connection_parameters=milvus_connection_parameters,\n",
    "    search_parameters=search_parameters,\n",
    "    collection_load_parameters=collection_load_parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "ea9d84f7-d142-4afa-9a6f-6c310d9604b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Union[Tuple[apache_beam.pvalue.PCollection[~MLTransformOutputT], apache_beam.pvalue.PCollection[apache_beam.pvalue.Row]], apache_beam.pvalue.PCollection[~MLTransformOutputT]] instead.\n",
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Union[Tuple[apache_beam.pvalue.PCollection[~MLTransformOutputT], apache_beam.pvalue.PCollection[apache_beam.pvalue.Row]], apache_beam.pvalue.PCollection[~MLTransformOutputT]] instead.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"query\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"real-time data processing system\"</span>,\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"query_embedding\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span>,\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"results\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "    <span style=\"font-weight: bold\">{</span>\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"id\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"1_0\"</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"distance\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.45</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"fields\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"content\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam is an open-source framework that provides a consistent programming model for both batch and streaming data processing. Developed originally by Google, it allows developers to write pipelines that can run on multiple engines, such as Apache Flink, Spark, and Google Cloud Dataflow. Beam uses abstractions like PCollections (data containers) and PTransforms (operations) to define the flow of data. The framework promotes portability through its runner architecture, letting the same pipeline execute on different backends. Support for multiple SDKs, including Java and Python, makes it accessible for a broad audience. Key features include support for event time, windowing, triggers, and stateful processing, which are essential for handling real-time data effectively. Beam is ideal for building ETL jobs, real-time analytics, and machine learning data pipelines. It helps teams focus on logic rather than infrastructure, offering flexibility and scalability in handling unbounded and bounded data sources. Apache Beam also supports a wide range of connectors for both input and output, including Kafka, BigQuery, and JDBC-based systems. This makes it easy to integrate Beam into existing data ecosystems. Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing.\"</span>,\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"metadata\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"title\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam: Unified Model for Batch and Streaming Data\"</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"keywords\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"stream processing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"batch processing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"data pipelines\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"SDK\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"tags\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Data Engineering\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Open Source\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Streaming\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Batch\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Big Data\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "      <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>,\n",
       "    <span style=\"font-weight: bold\">{</span>\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"id\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"2_1\"</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"distance\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.3929</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"fields\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"content\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"For developers, Dataflow provides local testing capabilities and a unified logging system through Cloud Logging. It also supports SQL-based pipeline definitions using BigQuery, which lowers the barrier to entry for analysts and data engineers. Dataflow’s streaming engine significantly improves performance and reduces costs by decoupling compute and state management. In summary, Google Cloud Dataflow not only simplifies the deployment of Apache Beam pipelines but also enhances them with cloud-native features. Its managed runtime, high availability, and integration with the broader Google Cloud ecosystem make it a powerful tool for modern data processing.\"</span>,\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"metadata\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"title\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud Dataflow: Run Apache Beam in the Cloud\"</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"keywords\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Dataflow\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"serverless\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"stream and batch\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"tags\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Cloud Computing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Data Pipelines\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Serverless\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Enterprise\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "      <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>\n",
       "  <span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"query\"\u001b[0m: \u001b[32m\"real-time data processing system\"\u001b[0m,\n",
       "  \u001b[1;34m\"query_embedding\"\u001b[0m: \u001b[1;36m384\u001b[0m,\n",
       "  \u001b[1;34m\"results\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "    \u001b[1m{\u001b[0m\n",
       "      \u001b[1;34m\"id\"\u001b[0m: \u001b[32m\"1_0\"\u001b[0m,\n",
       "      \u001b[1;34m\"distance\"\u001b[0m: \u001b[1;36m0.45\u001b[0m,\n",
       "      \u001b[1;34m\"fields\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[1;34m\"content\"\u001b[0m: \u001b[32m\"Apache Beam is an open-source framework that provides a consistent programming model for both batch and streaming data processing. Developed originally by Google, it allows developers to write pipelines that can run on multiple engines, such as Apache Flink, Spark, and Google Cloud Dataflow. Beam uses abstractions like PCollections (data containers) and PTransforms (operations) to define the flow of data. The framework promotes portability through its runner architecture, letting the same pipeline execute on different backends. Support for multiple SDKs, including Java and Python, makes it accessible for a broad audience. Key features include support for event time, windowing, triggers, and stateful processing, which are essential for handling real-time data effectively. Beam is ideal for building ETL jobs, real-time analytics, and machine learning data pipelines. It helps teams focus on logic rather than infrastructure, offering flexibility and scalability in handling unbounded and bounded data sources. Apache Beam also supports a wide range of connectors for both input and output, including Kafka, BigQuery, and JDBC-based systems. This makes it easy to integrate Beam into existing data ecosystems. Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing.\"\u001b[0m,\n",
       "        \u001b[1;34m\"metadata\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "          \u001b[1;34m\"title\"\u001b[0m: \u001b[32m\"Apache Beam: Unified Model for Batch and Streaming Data\"\u001b[0m,\n",
       "          \u001b[1;34m\"keywords\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Apache Beam\"\u001b[0m,\n",
       "            \u001b[32m\"stream processing\"\u001b[0m,\n",
       "            \u001b[32m\"batch processing\"\u001b[0m,\n",
       "            \u001b[32m\"data pipelines\"\u001b[0m,\n",
       "            \u001b[32m\"SDK\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m,\n",
       "          \u001b[1;34m\"tags\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Data Engineering\"\u001b[0m,\n",
       "            \u001b[32m\"Open Source\"\u001b[0m,\n",
       "            \u001b[32m\"Streaming\"\u001b[0m,\n",
       "            \u001b[32m\"Batch\"\u001b[0m,\n",
       "            \u001b[32m\"Big Data\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "      \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m,\n",
       "    \u001b[1m{\u001b[0m\n",
       "      \u001b[1;34m\"id\"\u001b[0m: \u001b[32m\"2_1\"\u001b[0m,\n",
       "      \u001b[1;34m\"distance\"\u001b[0m: \u001b[1;36m0.3929\u001b[0m,\n",
       "      \u001b[1;34m\"fields\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[1;34m\"content\"\u001b[0m: \u001b[32m\"For developers, Dataflow provides local testing capabilities and a unified logging system through Cloud Logging. It also supports SQL-based pipeline definitions using BigQuery, which lowers the barrier to entry for analysts and data engineers. Dataflow’s streaming engine significantly improves performance and reduces costs by decoupling compute and state management. In summary, Google Cloud Dataflow not only simplifies the deployment of Apache Beam pipelines but also enhances them with cloud-native features. Its managed runtime, high availability, and integration with the broader Google Cloud ecosystem make it a powerful tool for modern data processing.\"\u001b[0m,\n",
       "        \u001b[1;34m\"metadata\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "          \u001b[1;34m\"title\"\u001b[0m: \u001b[32m\"Google Cloud Dataflow: Run Apache Beam in the Cloud\"\u001b[0m,\n",
       "          \u001b[1;34m\"keywords\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Google Cloud\"\u001b[0m,\n",
       "            \u001b[32m\"Dataflow\"\u001b[0m,\n",
       "            \u001b[32m\"Apache Beam\"\u001b[0m,\n",
       "            \u001b[32m\"serverless\"\u001b[0m,\n",
       "            \u001b[32m\"stream and batch\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m,\n",
       "          \u001b[1;34m\"tags\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Cloud Computing\"\u001b[0m,\n",
       "            \u001b[32m\"Data Pipelines\"\u001b[0m,\n",
       "            \u001b[32m\"Google Cloud\"\u001b[0m,\n",
       "            \u001b[32m\"Serverless\"\u001b[0m,\n",
       "            \u001b[32m\"Enterprise\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "      \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m\n",
       "  \u001b[1m]\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with beam.Pipeline() as p:\n",
    "    _ = (\n",
    "        p\n",
    "        | \"Creating Queries\" >> beam.Create([query_chunk])\n",
    "        | 'Generating Embeddings' >> MLTransform(\n",
    "            write_artifact_location=tempfile.mkdtemp()).with_transform(huggingface_embedder)\n",
    "        | \"Enriching W/ Milvus Hybrid Search\" >> Enrichment(milvus_handler)\n",
    "        | \"Formatting and Printing Results\" >> FormatAndPrintResults()\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58753d47-5e63-49ef-8d95-f9acd94b8c0e",
   "metadata": {},
   "source": [
    "### Filtered Search (Metadata Filtering)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fdd049f-e856-4fa8-b3df-1498b973946b",
   "metadata": {},
   "source": [
    "When a user queries `what is beam?` using a **vector search strategy**, the semantic nature of **vector embeddings** can lead to ambiguity. Without additional context, the system might confuse **Google Beam** (a 3D communication platform) with **Apache Beam** (a data processing framework).\n",
    "\n",
    "**Metadata filtering** directly solves this by adding contextual constraints. For instance, applying a **specific metadata filter** (e.g., `{\"category\": \"computing\"}` or `{\"domain\": \"communication\"}`) before the vector search ensures that only documents relevant to the intended concept are considered. This dramatically narrows down results, enhances search precision, and overcomes the limitations of pure content-based search by disambiguating terms like \"beam\" with specific, structured criteria."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c96898d-af2d-4401-a9ca-8d230fa95e6e",
   "metadata": {},
   "source": [
    "#### Without Filtered Search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e549b22-256e-44c8-9638-eafc3a844770",
   "metadata": {},
   "source": [
    "As seen in the search results down below when a user searches for `what is beam?` without applying filters, the search results include both `Apache Beam` and `Google Beam`. Filtered search can come in play here by limiting the relevant search results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "3d267853-649d-494f-bea6-bbfe20650f79",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"what is beam?\"\n",
    "query_chunk = Chunk(content=Content(text=query))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "28a45b1c-f9a5-452e-aea6-ac46f17e01bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "search_parameters = MilvusSearchParameters(\n",
    "    collection_name=collection_name,\n",
    "    search_strategy=VectorSearchParameters(\n",
    "        limit=10,\n",
    "        anns_field=\"embedding\",\n",
    "    ),\n",
    "    output_fields=[\"metadata\",\"content\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "9ce3f0c7-fd1d-49a1-81e9-b8153cd284ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "collection_load_parameters = MilvusCollectionLoadParameters() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "6fad29b5-c2b0-4458-ab83-b38eb15a7505",
   "metadata": {},
   "outputs": [],
   "source": [
    "milvus_handler = MilvusSearchEnrichmentHandler(\n",
    "    connection_parameters=milvus_connection_parameters,\n",
    "    search_parameters=search_parameters,\n",
    "    collection_load_parameters=collection_load_parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "77add8a8-ddb8-48de-b1af-632d78c0d112",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Union[Tuple[apache_beam.pvalue.PCollection[~MLTransformOutputT], apache_beam.pvalue.PCollection[apache_beam.pvalue.Row]], apache_beam.pvalue.PCollection[~MLTransformOutputT]] instead.\n",
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Union[Tuple[apache_beam.pvalue.PCollection[~MLTransformOutputT], apache_beam.pvalue.PCollection[apache_beam.pvalue.Row]], apache_beam.pvalue.PCollection[~MLTransformOutputT]] instead.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"query\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"what is beam?\"</span>,\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"query_embedding\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span>,\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"results\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "    <span style=\"font-weight: bold\">{</span>\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"id\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"1_0\"</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"distance\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.4598</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"fields\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"content\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam is an open-source framework that provides a consistent programming model for both batch and streaming data processing. Developed originally by Google, it allows developers to write pipelines that can run on multiple engines, such as Apache Flink, Spark, and Google Cloud Dataflow. Beam uses abstractions like PCollections (data containers) and PTransforms (operations) to define the flow of data. The framework promotes portability through its runner architecture, letting the same pipeline execute on different backends. Support for multiple SDKs, including Java and Python, makes it accessible for a broad audience. Key features include support for event time, windowing, triggers, and stateful processing, which are essential for handling real-time data effectively. Beam is ideal for building ETL jobs, real-time analytics, and machine learning data pipelines. It helps teams focus on logic rather than infrastructure, offering flexibility and scalability in handling unbounded and bounded data sources. Apache Beam also supports a wide range of connectors for both input and output, including Kafka, BigQuery, and JDBC-based systems. This makes it easy to integrate Beam into existing data ecosystems. Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing.\"</span>,\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"metadata\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"title\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam: Unified Model for Batch and Streaming Data\"</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"keywords\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"stream processing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"batch processing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"data pipelines\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"SDK\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"tags\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Data Engineering\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Open Source\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Streaming\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Batch\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Big Data\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "      <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>,\n",
       "    <span style=\"font-weight: bold\">{</span>\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"id\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"1_1\"</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"distance\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.4353</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"fields\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"content\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing. The concept of runners enables developers to write once and run anywhere, which is particularly appealing for organizations that want to avoid vendor lock-in. The Beam model is based on a unified programming model that decouples pipeline logic from execution. This makes it easier to reason about time and state in both batch and streaming pipelines. Advanced features like late data handling, watermarks, and session windowing allow for more accurate and meaningful processing of real-world data. Beam also integrates with orchestration tools and monitoring systems, allowing for production-grade deployments. Community support and contributions have grown significantly, making Beam a stable and evolving ecosystem. Many cloud providers offer native support for Beam pipelines, and it's increasingly a core component in modern data platform architectures.\"</span>,\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"metadata\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"title\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam: Unified Model for Batch and Streaming Data\"</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"keywords\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"stream processing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"batch processing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"data pipelines\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"SDK\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"tags\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Data Engineering\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Open Source\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Streaming\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Batch\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Big Data\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "      <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>,\n",
       "    <span style=\"font-weight: bold\">{</span>\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"id\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"3_0\"</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"distance\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.3927</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"fields\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"content\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Beam is an innovative video communication platform that builds on the research of Project Starline. It uses AI, 3D imaging, and light field rendering to create immersive, lifelike video calls. Designed to replicate in-person interaction, Beam allows users to see life-sized, three-dimensional representations of each other without the need for headsets. This breakthrough makes remote conversations feel natural—capturing facial expressions, eye contact, and subtle gestures that traditional video conferencing often misses. Beam reduces meeting fatigue and enhances engagement, making it ideal for enterprise collaboration, interviews, and virtual presence scenarios. Powered by Google AI, Beam represents a significant leap in communication technology. Major companies like Salesforce, Deloitte, and NEC are already exploring its impact on digital collaboration. Google is partnering with HP to build and distribute Beam hardware, designed to work with existing productivity and video tools. Currently in limited early access for enterprise partners, Google Beam aims to redefine virtual meetings by bridging the gap between digital and physical presence. It’s a promising step toward more human and effective remote interactions.\"</span>,\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"metadata\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"title\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Beam: 3D Communication Powered by AI\"</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"keywords\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Beam\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Project Starline\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"3D video\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"AI communication\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"real-time meetings\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"tags\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"AI\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Communication\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"3D Technology\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Remote Work\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Enterprise Tech\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "      <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>,\n",
       "    <span style=\"font-weight: bold\">{</span>\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"id\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"2_1\"</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"distance\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.2925</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"fields\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"content\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"For developers, Dataflow provides local testing capabilities and a unified logging system through Cloud Logging. It also supports SQL-based pipeline definitions using BigQuery, which lowers the barrier to entry for analysts and data engineers. Dataflow’s streaming engine significantly improves performance and reduces costs by decoupling compute and state management. In summary, Google Cloud Dataflow not only simplifies the deployment of Apache Beam pipelines but also enhances them with cloud-native features. Its managed runtime, high availability, and integration with the broader Google Cloud ecosystem make it a powerful tool for modern data processing.\"</span>,\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"metadata\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"title\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud Dataflow: Run Apache Beam in the Cloud\"</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"keywords\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Dataflow\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"serverless\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"stream and batch\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"tags\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Cloud Computing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Data Pipelines\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Serverless\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Enterprise\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "      <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>,\n",
       "    <span style=\"font-weight: bold\">{</span>\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"id\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"2_0\"</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"distance\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.2342</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"fields\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"content\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud Dataflow is a fully managed service that runs Apache Beam pipelines in the cloud. It abstracts away infrastructure management and handles dynamic scaling, load balancing, and fault tolerance. Developers can focus on writing data logic using the Beam SDK and deploy it easily to Google Cloud. Dataflow supports both batch and stream processing and integrates seamlessly with other Google services like BigQuery, Pub/Sub, and Cloud Storage. Its autoscaling capabilities allow it to adapt to changing data volumes, optimizing for cost and performance. Features like monitoring dashboards, job templates, and built-in logging make it suitable for both development and production use. With support for event time processing, stateful functions, and windowing, Dataflow is well-suited for real-time analytics and data transformation tasks. It’s a key component for architects building scalable, cloud-native data platforms. Dataflow also offers templates for common ETL tasks, helping teams get started quickly with minimal setup. Its integration with Cloud Functions and Cloud Composer enables event-driven and orchestrated workflows. Security and compliance are built-in with IAM roles, encryption at rest and in transit, and audit logging, making it suitable for enterprise environments.\"</span>,\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"metadata\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"title\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud Dataflow: Run Apache Beam in the Cloud\"</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"keywords\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Dataflow\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"serverless\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"stream and batch\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"tags\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Cloud Computing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Data Pipelines\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Cloud\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Serverless\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Enterprise\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "      <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>\n",
       "  <span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"query\"\u001b[0m: \u001b[32m\"what is beam?\"\u001b[0m,\n",
       "  \u001b[1;34m\"query_embedding\"\u001b[0m: \u001b[1;36m384\u001b[0m,\n",
       "  \u001b[1;34m\"results\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "    \u001b[1m{\u001b[0m\n",
       "      \u001b[1;34m\"id\"\u001b[0m: \u001b[32m\"1_0\"\u001b[0m,\n",
       "      \u001b[1;34m\"distance\"\u001b[0m: \u001b[1;36m0.4598\u001b[0m,\n",
       "      \u001b[1;34m\"fields\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[1;34m\"content\"\u001b[0m: \u001b[32m\"Apache Beam is an open-source framework that provides a consistent programming model for both batch and streaming data processing. Developed originally by Google, it allows developers to write pipelines that can run on multiple engines, such as Apache Flink, Spark, and Google Cloud Dataflow. Beam uses abstractions like PCollections (data containers) and PTransforms (operations) to define the flow of data. The framework promotes portability through its runner architecture, letting the same pipeline execute on different backends. Support for multiple SDKs, including Java and Python, makes it accessible for a broad audience. Key features include support for event time, windowing, triggers, and stateful processing, which are essential for handling real-time data effectively. Beam is ideal for building ETL jobs, real-time analytics, and machine learning data pipelines. It helps teams focus on logic rather than infrastructure, offering flexibility and scalability in handling unbounded and bounded data sources. Apache Beam also supports a wide range of connectors for both input and output, including Kafka, BigQuery, and JDBC-based systems. This makes it easy to integrate Beam into existing data ecosystems. Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing.\"\u001b[0m,\n",
       "        \u001b[1;34m\"metadata\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "          \u001b[1;34m\"title\"\u001b[0m: \u001b[32m\"Apache Beam: Unified Model for Batch and Streaming Data\"\u001b[0m,\n",
       "          \u001b[1;34m\"keywords\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Apache Beam\"\u001b[0m,\n",
       "            \u001b[32m\"stream processing\"\u001b[0m,\n",
       "            \u001b[32m\"batch processing\"\u001b[0m,\n",
       "            \u001b[32m\"data pipelines\"\u001b[0m,\n",
       "            \u001b[32m\"SDK\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m,\n",
       "          \u001b[1;34m\"tags\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Data Engineering\"\u001b[0m,\n",
       "            \u001b[32m\"Open Source\"\u001b[0m,\n",
       "            \u001b[32m\"Streaming\"\u001b[0m,\n",
       "            \u001b[32m\"Batch\"\u001b[0m,\n",
       "            \u001b[32m\"Big Data\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "      \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m,\n",
       "    \u001b[1m{\u001b[0m\n",
       "      \u001b[1;34m\"id\"\u001b[0m: \u001b[32m\"1_1\"\u001b[0m,\n",
       "      \u001b[1;34m\"distance\"\u001b[0m: \u001b[1;36m0.4353\u001b[0m,\n",
       "      \u001b[1;34m\"fields\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[1;34m\"content\"\u001b[0m: \u001b[32m\"Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing. The concept of runners enables developers to write once and run anywhere, which is particularly appealing for organizations that want to avoid vendor lock-in. The Beam model is based on a unified programming model that decouples pipeline logic from execution. This makes it easier to reason about time and state in both batch and streaming pipelines. Advanced features like late data handling, watermarks, and session windowing allow for more accurate and meaningful processing of real-world data. Beam also integrates with orchestration tools and monitoring systems, allowing for production-grade deployments. Community support and contributions have grown significantly, making Beam a stable and evolving ecosystem. Many cloud providers offer native support for Beam pipelines, and it's increasingly a core component in modern data platform architectures.\"\u001b[0m,\n",
       "        \u001b[1;34m\"metadata\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "          \u001b[1;34m\"title\"\u001b[0m: \u001b[32m\"Apache Beam: Unified Model for Batch and Streaming Data\"\u001b[0m,\n",
       "          \u001b[1;34m\"keywords\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Apache Beam\"\u001b[0m,\n",
       "            \u001b[32m\"stream processing\"\u001b[0m,\n",
       "            \u001b[32m\"batch processing\"\u001b[0m,\n",
       "            \u001b[32m\"data pipelines\"\u001b[0m,\n",
       "            \u001b[32m\"SDK\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m,\n",
       "          \u001b[1;34m\"tags\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Data Engineering\"\u001b[0m,\n",
       "            \u001b[32m\"Open Source\"\u001b[0m,\n",
       "            \u001b[32m\"Streaming\"\u001b[0m,\n",
       "            \u001b[32m\"Batch\"\u001b[0m,\n",
       "            \u001b[32m\"Big Data\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "      \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m,\n",
       "    \u001b[1m{\u001b[0m\n",
       "      \u001b[1;34m\"id\"\u001b[0m: \u001b[32m\"3_0\"\u001b[0m,\n",
       "      \u001b[1;34m\"distance\"\u001b[0m: \u001b[1;36m0.3927\u001b[0m,\n",
       "      \u001b[1;34m\"fields\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[1;34m\"content\"\u001b[0m: \u001b[32m\"Google Beam is an innovative video communication platform that builds on the research of Project Starline. It uses AI, 3D imaging, and light field rendering to create immersive, lifelike video calls. Designed to replicate in-person interaction, Beam allows users to see life-sized, three-dimensional representations of each other without the need for headsets. This breakthrough makes remote conversations feel natural—capturing facial expressions, eye contact, and subtle gestures that traditional video conferencing often misses. Beam reduces meeting fatigue and enhances engagement, making it ideal for enterprise collaboration, interviews, and virtual presence scenarios. Powered by Google AI, Beam represents a significant leap in communication technology. Major companies like Salesforce, Deloitte, and NEC are already exploring its impact on digital collaboration. Google is partnering with HP to build and distribute Beam hardware, designed to work with existing productivity and video tools. Currently in limited early access for enterprise partners, Google Beam aims to redefine virtual meetings by bridging the gap between digital and physical presence. It’s a promising step toward more human and effective remote interactions.\"\u001b[0m,\n",
       "        \u001b[1;34m\"metadata\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "          \u001b[1;34m\"title\"\u001b[0m: \u001b[32m\"Google Beam: 3D Communication Powered by AI\"\u001b[0m,\n",
       "          \u001b[1;34m\"keywords\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Google Beam\"\u001b[0m,\n",
       "            \u001b[32m\"Project Starline\"\u001b[0m,\n",
       "            \u001b[32m\"3D video\"\u001b[0m,\n",
       "            \u001b[32m\"AI communication\"\u001b[0m,\n",
       "            \u001b[32m\"real-time meetings\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m,\n",
       "          \u001b[1;34m\"tags\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"AI\"\u001b[0m,\n",
       "            \u001b[32m\"Communication\"\u001b[0m,\n",
       "            \u001b[32m\"3D Technology\"\u001b[0m,\n",
       "            \u001b[32m\"Remote Work\"\u001b[0m,\n",
       "            \u001b[32m\"Enterprise Tech\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "      \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m,\n",
       "    \u001b[1m{\u001b[0m\n",
       "      \u001b[1;34m\"id\"\u001b[0m: \u001b[32m\"2_1\"\u001b[0m,\n",
       "      \u001b[1;34m\"distance\"\u001b[0m: \u001b[1;36m0.2925\u001b[0m,\n",
       "      \u001b[1;34m\"fields\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[1;34m\"content\"\u001b[0m: \u001b[32m\"For developers, Dataflow provides local testing capabilities and a unified logging system through Cloud Logging. It also supports SQL-based pipeline definitions using BigQuery, which lowers the barrier to entry for analysts and data engineers. Dataflow’s streaming engine significantly improves performance and reduces costs by decoupling compute and state management. In summary, Google Cloud Dataflow not only simplifies the deployment of Apache Beam pipelines but also enhances them with cloud-native features. Its managed runtime, high availability, and integration with the broader Google Cloud ecosystem make it a powerful tool for modern data processing.\"\u001b[0m,\n",
       "        \u001b[1;34m\"metadata\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "          \u001b[1;34m\"title\"\u001b[0m: \u001b[32m\"Google Cloud Dataflow: Run Apache Beam in the Cloud\"\u001b[0m,\n",
       "          \u001b[1;34m\"keywords\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Google Cloud\"\u001b[0m,\n",
       "            \u001b[32m\"Dataflow\"\u001b[0m,\n",
       "            \u001b[32m\"Apache Beam\"\u001b[0m,\n",
       "            \u001b[32m\"serverless\"\u001b[0m,\n",
       "            \u001b[32m\"stream and batch\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m,\n",
       "          \u001b[1;34m\"tags\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Cloud Computing\"\u001b[0m,\n",
       "            \u001b[32m\"Data Pipelines\"\u001b[0m,\n",
       "            \u001b[32m\"Google Cloud\"\u001b[0m,\n",
       "            \u001b[32m\"Serverless\"\u001b[0m,\n",
       "            \u001b[32m\"Enterprise\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "      \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m,\n",
       "    \u001b[1m{\u001b[0m\n",
       "      \u001b[1;34m\"id\"\u001b[0m: \u001b[32m\"2_0\"\u001b[0m,\n",
       "      \u001b[1;34m\"distance\"\u001b[0m: \u001b[1;36m0.2342\u001b[0m,\n",
       "      \u001b[1;34m\"fields\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[1;34m\"content\"\u001b[0m: \u001b[32m\"Google Cloud Dataflow is a fully managed service that runs Apache Beam pipelines in the cloud. It abstracts away infrastructure management and handles dynamic scaling, load balancing, and fault tolerance. Developers can focus on writing data logic using the Beam SDK and deploy it easily to Google Cloud. Dataflow supports both batch and stream processing and integrates seamlessly with other Google services like BigQuery, Pub/Sub, and Cloud Storage. Its autoscaling capabilities allow it to adapt to changing data volumes, optimizing for cost and performance. Features like monitoring dashboards, job templates, and built-in logging make it suitable for both development and production use. With support for event time processing, stateful functions, and windowing, Dataflow is well-suited for real-time analytics and data transformation tasks. It’s a key component for architects building scalable, cloud-native data platforms. Dataflow also offers templates for common ETL tasks, helping teams get started quickly with minimal setup. Its integration with Cloud Functions and Cloud Composer enables event-driven and orchestrated workflows. Security and compliance are built-in with IAM roles, encryption at rest and in transit, and audit logging, making it suitable for enterprise environments.\"\u001b[0m,\n",
       "        \u001b[1;34m\"metadata\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "          \u001b[1;34m\"title\"\u001b[0m: \u001b[32m\"Google Cloud Dataflow: Run Apache Beam in the Cloud\"\u001b[0m,\n",
       "          \u001b[1;34m\"keywords\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Google Cloud\"\u001b[0m,\n",
       "            \u001b[32m\"Dataflow\"\u001b[0m,\n",
       "            \u001b[32m\"Apache Beam\"\u001b[0m,\n",
       "            \u001b[32m\"serverless\"\u001b[0m,\n",
       "            \u001b[32m\"stream and batch\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m,\n",
       "          \u001b[1;34m\"tags\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Cloud Computing\"\u001b[0m,\n",
       "            \u001b[32m\"Data Pipelines\"\u001b[0m,\n",
       "            \u001b[32m\"Google Cloud\"\u001b[0m,\n",
       "            \u001b[32m\"Serverless\"\u001b[0m,\n",
       "            \u001b[32m\"Enterprise\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "      \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m\n",
       "  \u001b[1m]\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with beam.Pipeline() as p:\n",
    "    _ = (\n",
    "        p\n",
    "        | \"Creating Queries\" >> beam.Create([query_chunk])\n",
    "        | 'Generating Embeddings' >> MLTransform(\n",
    "            write_artifact_location=tempfile.mkdtemp()).with_transform(huggingface_embedder)\n",
    "        | \"Enriching W/ Milvus Vector Search\" >> Enrichment(milvus_handler)\n",
    "        | \"Formatting and Printing Results\" >> FormatAndPrintResults())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb72f9c6-5a29-4810-9768-574aa7ea5128",
   "metadata": {},
   "source": [
    "#### Searching for Apache Beam with Filtered Search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df64b70f-bad8-469f-8419-723911f7f7cf",
   "metadata": {},
   "source": [
    "To precisely target **Apache Beam** and ensure the retrieval of only relevant documents, we can leverage the power of **metadata filtering**. By applying a filter that specifies the document's `keywords` must contain `data pipelines`, we can instruct the undelrying search engine to exclude any documents related to `Google Beam` from the result set. This allows the vector search to operate on a pre-filtered, highly relevant subset of the corpus, guaranteeing that the retrieved information pertains exclusively to `Apache Beam`'s domain, thereby resolving the semantic ambiguity with remarkable precision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "6e79ef5c-a121-4e69-9089-0991821f8745",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"what is beam?\"\n",
    "query_chunk = Chunk(content=Content(text=query))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "5314c531-14bb-4d81-92a5-fcf9cca7fa81",
   "metadata": {},
   "outputs": [],
   "source": [
    "search_parameters = MilvusSearchParameters(\n",
    "    collection_name=collection_name,\n",
    "    search_strategy=VectorSearchParameters(\n",
    "        filter=\"ARRAY_CONTAINS(metadata['keywords'], 'data pipelines')\",\n",
    "        limit=10,\n",
    "        anns_field=\"embedding\",\n",
    "    ),\n",
    "    output_fields=[\"metadata\",\"content\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "0ecf2ac6-cf90-4ce7-b17f-113af90ab950",
   "metadata": {},
   "outputs": [],
   "source": [
    "collection_load_parameters = MilvusCollectionLoadParameters() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "0cd92b69-b9dc-445c-9bd7-21bb3ceb0fd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "milvus_handler = MilvusSearchEnrichmentHandler(\n",
    "    connection_parameters=milvus_connection_parameters,\n",
    "    search_parameters=search_parameters,\n",
    "    collection_load_parameters=collection_load_parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "b06ecf64-c314-4c6a-ae1a-4fdf059aeead",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Union[Tuple[apache_beam.pvalue.PCollection[~MLTransformOutputT], apache_beam.pvalue.PCollection[apache_beam.pvalue.Row]], apache_beam.pvalue.PCollection[~MLTransformOutputT]] instead.\n",
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Union[Tuple[apache_beam.pvalue.PCollection[~MLTransformOutputT], apache_beam.pvalue.PCollection[apache_beam.pvalue.Row]], apache_beam.pvalue.PCollection[~MLTransformOutputT]] instead.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"query\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"what is beam?\"</span>,\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"query_embedding\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span>,\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"results\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "    <span style=\"font-weight: bold\">{</span>\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"id\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"1_0\"</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"distance\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.4598</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"fields\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"content\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam is an open-source framework that provides a consistent programming model for both batch and streaming data processing. Developed originally by Google, it allows developers to write pipelines that can run on multiple engines, such as Apache Flink, Spark, and Google Cloud Dataflow. Beam uses abstractions like PCollections (data containers) and PTransforms (operations) to define the flow of data. The framework promotes portability through its runner architecture, letting the same pipeline execute on different backends. Support for multiple SDKs, including Java and Python, makes it accessible for a broad audience. Key features include support for event time, windowing, triggers, and stateful processing, which are essential for handling real-time data effectively. Beam is ideal for building ETL jobs, real-time analytics, and machine learning data pipelines. It helps teams focus on logic rather than infrastructure, offering flexibility and scalability in handling unbounded and bounded data sources. Apache Beam also supports a wide range of connectors for both input and output, including Kafka, BigQuery, and JDBC-based systems. This makes it easy to integrate Beam into existing data ecosystems. Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing.\"</span>,\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"metadata\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"title\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam: Unified Model for Batch and Streaming Data\"</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"keywords\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"stream processing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"batch processing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"data pipelines\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"SDK\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"tags\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Data Engineering\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Open Source\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Streaming\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Batch\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Big Data\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "      <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>,\n",
       "    <span style=\"font-weight: bold\">{</span>\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"id\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"1_1\"</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"distance\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.4353</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"fields\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"content\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing. The concept of runners enables developers to write once and run anywhere, which is particularly appealing for organizations that want to avoid vendor lock-in. The Beam model is based on a unified programming model that decouples pipeline logic from execution. This makes it easier to reason about time and state in both batch and streaming pipelines. Advanced features like late data handling, watermarks, and session windowing allow for more accurate and meaningful processing of real-world data. Beam also integrates with orchestration tools and monitoring systems, allowing for production-grade deployments. Community support and contributions have grown significantly, making Beam a stable and evolving ecosystem. Many cloud providers offer native support for Beam pipelines, and it's increasingly a core component in modern data platform architectures.\"</span>,\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"metadata\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"title\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam: Unified Model for Batch and Streaming Data\"</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"keywords\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Apache Beam\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"stream processing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"batch processing\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"data pipelines\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"SDK\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"tags\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Data Engineering\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Open Source\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Streaming\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Batch\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Big Data\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "      <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>\n",
       "  <span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"query\"\u001b[0m: \u001b[32m\"what is beam?\"\u001b[0m,\n",
       "  \u001b[1;34m\"query_embedding\"\u001b[0m: \u001b[1;36m384\u001b[0m,\n",
       "  \u001b[1;34m\"results\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "    \u001b[1m{\u001b[0m\n",
       "      \u001b[1;34m\"id\"\u001b[0m: \u001b[32m\"1_0\"\u001b[0m,\n",
       "      \u001b[1;34m\"distance\"\u001b[0m: \u001b[1;36m0.4598\u001b[0m,\n",
       "      \u001b[1;34m\"fields\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[1;34m\"content\"\u001b[0m: \u001b[32m\"Apache Beam is an open-source framework that provides a consistent programming model for both batch and streaming data processing. Developed originally by Google, it allows developers to write pipelines that can run on multiple engines, such as Apache Flink, Spark, and Google Cloud Dataflow. Beam uses abstractions like PCollections (data containers) and PTransforms (operations) to define the flow of data. The framework promotes portability through its runner architecture, letting the same pipeline execute on different backends. Support for multiple SDKs, including Java and Python, makes it accessible for a broad audience. Key features include support for event time, windowing, triggers, and stateful processing, which are essential for handling real-time data effectively. Beam is ideal for building ETL jobs, real-time analytics, and machine learning data pipelines. It helps teams focus on logic rather than infrastructure, offering flexibility and scalability in handling unbounded and bounded data sources. Apache Beam also supports a wide range of connectors for both input and output, including Kafka, BigQuery, and JDBC-based systems. This makes it easy to integrate Beam into existing data ecosystems. Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing.\"\u001b[0m,\n",
       "        \u001b[1;34m\"metadata\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "          \u001b[1;34m\"title\"\u001b[0m: \u001b[32m\"Apache Beam: Unified Model for Batch and Streaming Data\"\u001b[0m,\n",
       "          \u001b[1;34m\"keywords\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Apache Beam\"\u001b[0m,\n",
       "            \u001b[32m\"stream processing\"\u001b[0m,\n",
       "            \u001b[32m\"batch processing\"\u001b[0m,\n",
       "            \u001b[32m\"data pipelines\"\u001b[0m,\n",
       "            \u001b[32m\"SDK\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m,\n",
       "          \u001b[1;34m\"tags\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Data Engineering\"\u001b[0m,\n",
       "            \u001b[32m\"Open Source\"\u001b[0m,\n",
       "            \u001b[32m\"Streaming\"\u001b[0m,\n",
       "            \u001b[32m\"Batch\"\u001b[0m,\n",
       "            \u001b[32m\"Big Data\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "      \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m,\n",
       "    \u001b[1m{\u001b[0m\n",
       "      \u001b[1;34m\"id\"\u001b[0m: \u001b[32m\"1_1\"\u001b[0m,\n",
       "      \u001b[1;34m\"distance\"\u001b[0m: \u001b[1;36m0.4353\u001b[0m,\n",
       "      \u001b[1;34m\"fields\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[1;34m\"content\"\u001b[0m: \u001b[32m\"Developers can build reusable transforms and modularize pipeline logic, improving maintainability and testing. The concept of runners enables developers to write once and run anywhere, which is particularly appealing for organizations that want to avoid vendor lock-in. The Beam model is based on a unified programming model that decouples pipeline logic from execution. This makes it easier to reason about time and state in both batch and streaming pipelines. Advanced features like late data handling, watermarks, and session windowing allow for more accurate and meaningful processing of real-world data. Beam also integrates with orchestration tools and monitoring systems, allowing for production-grade deployments. Community support and contributions have grown significantly, making Beam a stable and evolving ecosystem. Many cloud providers offer native support for Beam pipelines, and it's increasingly a core component in modern data platform architectures.\"\u001b[0m,\n",
       "        \u001b[1;34m\"metadata\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "          \u001b[1;34m\"title\"\u001b[0m: \u001b[32m\"Apache Beam: Unified Model for Batch and Streaming Data\"\u001b[0m,\n",
       "          \u001b[1;34m\"keywords\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Apache Beam\"\u001b[0m,\n",
       "            \u001b[32m\"stream processing\"\u001b[0m,\n",
       "            \u001b[32m\"batch processing\"\u001b[0m,\n",
       "            \u001b[32m\"data pipelines\"\u001b[0m,\n",
       "            \u001b[32m\"SDK\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m,\n",
       "          \u001b[1;34m\"tags\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Data Engineering\"\u001b[0m,\n",
       "            \u001b[32m\"Open Source\"\u001b[0m,\n",
       "            \u001b[32m\"Streaming\"\u001b[0m,\n",
       "            \u001b[32m\"Batch\"\u001b[0m,\n",
       "            \u001b[32m\"Big Data\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "      \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m\n",
       "  \u001b[1m]\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with beam.Pipeline() as p:\n",
    "    _ = (\n",
    "        p\n",
    "        | \"Creating Queries\" >> beam.Create([query_chunk])\n",
    "        | 'Generating Embeddings' >> MLTransform(\n",
    "            write_artifact_location=tempfile.mkdtemp()).with_transform(huggingface_embedder)\n",
    "        | \"Enriching W/ Milvus Vector Search\" >> Enrichment(milvus_handler)\n",
    "        | \"Formatting and Printing Results\" >> FormatAndPrintResults())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e61bcf4-96e7-47dd-bb37-4788e99a2b89",
   "metadata": {},
   "source": [
    "#### Searching for Google Beam with Filtered Search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a782f79b-a1a2-4474-807e-8abad62406b0",
   "metadata": {},
   "source": [
    "To precisely target `Google Beam` and ensure the retrieval of only relevant documents, we can leverage the power of `metadata filtering`. By applying a filter that specifies the document's `tags` must contain `Remote Work`, we can instruct the underlying search engine to exclude any documents related to `Apache Beam` from the result set. This allows the vector search to operate on a pre-filtered, highly relevant subset of the corpus, guaranteeing that the retrieved information pertains exclusively to `Google Beam`'s domain, thereby resolving the semantic ambiguity with remarkable precision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "a8077395-c374-400f-abdc-fe6630eab8a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"what is beam?\"\n",
    "query_chunk = Chunk(content=Content(text=query))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "3b712779-f283-4e37-88ed-d6b65c6c45d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "search_parameters = MilvusSearchParameters(\n",
    "    collection_name=collection_name,\n",
    "    search_strategy=VectorSearchParameters(filter=\"ARRAY_CONTAINS(metadata['tags'], 'Remote Work')\",limit=10,anns_field=\"embedding\"),\n",
    "    output_fields=[\"metadata\", \"content\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "7f0924a3-8832-4138-a599-d3aef648b962",
   "metadata": {},
   "outputs": [],
   "source": [
    "collection_load_parameters = MilvusCollectionLoadParameters() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "516ecbf0-9bb0-4177-829b-b79300b29bbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "milvus_handler = MilvusSearchEnrichmentHandler(\n",
    "    connection_parameters=milvus_connection_parameters,\n",
    "    search_parameters=search_parameters,\n",
    "    collection_load_parameters=collection_load_parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "db32dda5-0668-4162-80ea-b6a0c2a79063",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Union[Tuple[apache_beam.pvalue.PCollection[~MLTransformOutputT], apache_beam.pvalue.PCollection[apache_beam.pvalue.Row]], apache_beam.pvalue.PCollection[~MLTransformOutputT]] instead.\n",
      "WARNING:root:This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Union[Tuple[apache_beam.pvalue.PCollection[~MLTransformOutputT], apache_beam.pvalue.PCollection[apache_beam.pvalue.Row]], apache_beam.pvalue.PCollection[~MLTransformOutputT]] instead.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"query\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"what is beam?\"</span>,\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"query_embedding\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span>,\n",
       "  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"results\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "    <span style=\"font-weight: bold\">{</span>\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"id\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"3_0\"</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"distance\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.3927</span>,\n",
       "      <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"fields\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"content\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Beam is an innovative video communication platform that builds on the research of Project Starline. It uses AI, 3D imaging, and light field rendering to create immersive, lifelike video calls. Designed to replicate in-person interaction, Beam allows users to see life-sized, three-dimensional representations of each other without the need for headsets. This breakthrough makes remote conversations feel natural—capturing facial expressions, eye contact, and subtle gestures that traditional video conferencing often misses. Beam reduces meeting fatigue and enhances engagement, making it ideal for enterprise collaboration, interviews, and virtual presence scenarios. Powered by Google AI, Beam represents a significant leap in communication technology. Major companies like Salesforce, Deloitte, and NEC are already exploring its impact on digital collaboration. Google is partnering with HP to build and distribute Beam hardware, designed to work with existing productivity and video tools. Currently in limited early access for enterprise partners, Google Beam aims to redefine virtual meetings by bridging the gap between digital and physical presence. It’s a promising step toward more human and effective remote interactions.\"</span>,\n",
       "        <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"metadata\"</span>: <span style=\"font-weight: bold\">{</span>\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"title\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Beam: 3D Communication Powered by AI\"</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"keywords\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Google Beam\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Project Starline\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"3D video\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"AI communication\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"real-time meetings\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>,\n",
       "          <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">\"tags\"</span>: <span style=\"font-weight: bold\">[</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"AI\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Communication\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"3D Technology\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Remote Work\"</span>,\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">\"Enterprise Tech\"</span>\n",
       "          <span style=\"font-weight: bold\">]</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "      <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>\n",
       "  <span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "  \u001b[1;34m\"query\"\u001b[0m: \u001b[32m\"what is beam?\"\u001b[0m,\n",
       "  \u001b[1;34m\"query_embedding\"\u001b[0m: \u001b[1;36m384\u001b[0m,\n",
       "  \u001b[1;34m\"results\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "    \u001b[1m{\u001b[0m\n",
       "      \u001b[1;34m\"id\"\u001b[0m: \u001b[32m\"3_0\"\u001b[0m,\n",
       "      \u001b[1;34m\"distance\"\u001b[0m: \u001b[1;36m0.3927\u001b[0m,\n",
       "      \u001b[1;34m\"fields\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[1;34m\"content\"\u001b[0m: \u001b[32m\"Google Beam is an innovative video communication platform that builds on the research of Project Starline. It uses AI, 3D imaging, and light field rendering to create immersive, lifelike video calls. Designed to replicate in-person interaction, Beam allows users to see life-sized, three-dimensional representations of each other without the need for headsets. This breakthrough makes remote conversations feel natural—capturing facial expressions, eye contact, and subtle gestures that traditional video conferencing often misses. Beam reduces meeting fatigue and enhances engagement, making it ideal for enterprise collaboration, interviews, and virtual presence scenarios. Powered by Google AI, Beam represents a significant leap in communication technology. Major companies like Salesforce, Deloitte, and NEC are already exploring its impact on digital collaboration. Google is partnering with HP to build and distribute Beam hardware, designed to work with existing productivity and video tools. Currently in limited early access for enterprise partners, Google Beam aims to redefine virtual meetings by bridging the gap between digital and physical presence. It’s a promising step toward more human and effective remote interactions.\"\u001b[0m,\n",
       "        \u001b[1;34m\"metadata\"\u001b[0m: \u001b[1m{\u001b[0m\n",
       "          \u001b[1;34m\"title\"\u001b[0m: \u001b[32m\"Google Beam: 3D Communication Powered by AI\"\u001b[0m,\n",
       "          \u001b[1;34m\"keywords\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"Google Beam\"\u001b[0m,\n",
       "            \u001b[32m\"Project Starline\"\u001b[0m,\n",
       "            \u001b[32m\"3D video\"\u001b[0m,\n",
       "            \u001b[32m\"AI communication\"\u001b[0m,\n",
       "            \u001b[32m\"real-time meetings\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m,\n",
       "          \u001b[1;34m\"tags\"\u001b[0m: \u001b[1m[\u001b[0m\n",
       "            \u001b[32m\"AI\"\u001b[0m,\n",
       "            \u001b[32m\"Communication\"\u001b[0m,\n",
       "            \u001b[32m\"3D Technology\"\u001b[0m,\n",
       "            \u001b[32m\"Remote Work\"\u001b[0m,\n",
       "            \u001b[32m\"Enterprise Tech\"\u001b[0m\n",
       "          \u001b[1m]\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "      \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m\n",
       "  \u001b[1m]\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with beam.Pipeline() as p:\n",
    "    _ = (\n",
    "        p\n",
    "        | \"Creating Queries\" >> beam.Create([query_chunk])\n",
    "        | 'Generating Embeddings' >> MLTransform(\n",
    "            write_artifact_location=tempfile.mkdtemp()).with_transform(huggingface_embedder)\n",
    "        | \"Enriching W/ Milvus Vector Search\" >> Enrichment(milvus_handler)\n",
    "        | \"Formatting and Printing Results\" >> FormatAndPrintResults())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2670682-24bf-45b6-9593-bed0e3b1cee2",
   "metadata": {},
   "source": [
    "## Cleanup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "0a3f4d66-3823-46c7-8a58-e9e8ac7899c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "MilvusEnrichmentTestHelper.stop_db_container(db)\n",
    "db = None"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.24"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}