Automating Internet Search Utilizing LangChain and Google Search APIs

Introduction 

Synthetic intelligence is increasing within the fashionable world as a result of to a large number of research and innovations within the discipline from varied startups and organizations. Researchers and innovators are creating a variety of instruments and know-how to help the creation of LLM-powered functions. With the help of AI and NLP improvements like LangChain and LLMs, customers can get across the limitations of conventional search strategies, akin to having to comb by way of dozens of hyperlinks and web sites to search out rLLMselevant info. As a substitute, customers can use search engine APIs like Google Search APIs and LangChain and OpenAI to obtain a concise, summarized response to their question together with hyperlinks to associated assets.

On this article, we’re going to learn the way modern framework like LangChain with Google search APls can create net automation software to ask and get solutions from the knowledge retrieved info of huge array of net assets. This text covers fingers on information to construct net automation software from scratch to be used circumstances like analysis, evaluation, and so on. So let’s get began:

Studying Goals

• Study net scraping and automation software utilizing LangChain framework.
• Study a step-by-step information to construct an online automation software utilizing LangChain and Google Search APIs.
• Implement the combination of LangChain with Google Search APIs to automate net searches.

This text was revealed as part of the Information Science Blogathon.

What’s Internet Automation Software and its Workflow?

First, we’ll have a look at the standard net analysis and automation software workflow which is essential to know structure of such LLM powered functions. When person queries to the online analysis automation software, Google Search API takes question and returns variety of net hyperlinks that can be loaded utilizing net loader which scraps the online pages. loaded net content material will then be remodeled into readable textual content format by eradicating all undesirable html tags. Let’s look the under diagram for extra particulars:

Lastly, scraped and remodeled net pages content material can be loaded into vector shops akin to chroma, pinecone, FAISS, and so on for additional querying or Q&A for analysis function. Utilizing appropriate immediate engineering customers may also summarise the online content material for additional analysis and evaluation.

Internet Loader and Transformation

As soon as the person question brings the hyperlinks of the online content material relevent to the search question, it’ll thereafter be scraped utilizing “ChromiumLoader” or “HtmlLoader” to load the online content material into undertaking surroundings. as soon as the content material is loaded it’ll then be remodeled utilizing “BeautifulSoupTransformer” or “HtmltoTextTransformer” to take away html tags and get the online content material for additional processing. let’s have a look at the each the strategies with code examples for in depth understanding.

ChromiumLoader

Utilizing Playwright and python’s asyncio “ChromiumLoader” interacts with net pages within the browser to load the content material. thereafter, beautifulsoup transformer removes all of the html tags like <p>, <span>,<div>,<li>, and so on to extracts the textual content content material from it.

# set up obligatory libraries for the undertaking
!pip set up -q langchain-openai langchain playwright beautifulsoup4
!pip set up -q langchain_community

# scraping utilizing chromiumloader
from langchain_community.document_loaders import AsyncChromiumLoader
from langchain_community.document_transformers import BeautifulSoupTransformer

# Load HTML
loader = AsyncChromiumLoader(["https://www.wsj.com"], headless=True)

# load utilizing playwright
!playwright set up
html = await loader.aload()

# Rework the content material utilizing bs4 transformer
bs_transformer = BeautifulSoupTransformer()
documents_transformed = []
for doc in html:
    documents_transformed.prolong(bs_transformer.extract_tags(doc.page_content, 
                                                               tags="span"))

HtmlLoader

Equally, One other various for the online content material scraping is to make use of HtmlLoader which makes use of ‘aiohttp’ library to make async HTTP requests to scrape and cargo net pages.

# scrpe the online content material utilizing html loader
from langchain_community.document_loaders import AsyncHtmlLoader
!pip set up -q html2text

# load the content material 
urls = ["https://www.espn.com", "https://lilianweng.github.io/posts/2023-06-23-agent/"]
loader = AsyncHtmlLoader(urls)
docs = loader.load()

# html to textual content tranformation
from langchain_community.document_transformers import Html2TextTransformer

# Html to Textual content transformer
html2text = Html2TextTransformer()
documents_transformed = html2text.transform_documents(docs)
documents_transformed[0].page_content[0:500]

On this part, we’re going to be taught complete means of extracting the online content material from any given net and scrape it into desired construction utilizing massive language mannequin APIs. that is essential to search out summarised and correct solutions for the person question in net automation software. Let’s start by making a object for big language mannequin from OpenAI.

# assign OpenAI mannequin utilizing langchai OpenAI
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(temperature=0, mannequin="gpt-3.5-turbo-0613")

# outline a schema to extract content material from net web page
from langchain.chains import create_extraction_chain

schema = {
    "properties": {
        "news_article_title": {"sort": "string"},
        "news_article_summary": {"sort": "string"},
    },
    "required": ["news_article_title", "news_article_summary"],
}

# outline a extract perform to get summerized content material from the LLM name
def extract(content material: str, schema: dict):
    return create_extraction_chain(schema=schema, llm=llm).run(content material)

Above code takes LLM as enter together with schema of output within the extract perform. this perform can be known as as soon as the online content material is scraped and remodeled utilizing html loader and transformer.

# net scraper with bs4
import pprint

from langchain_text_splitters import RecursiveCharacterTextSplitter

# scrape the information utilizing playwright and htmlloader
def scrape_with_playwright(urls, schema):
    loader = AsyncHtmlLoader(urls)
    docs = loader.load()
    html_transformer = Html2TextTransformer()
    docs_transformed = html2text.transform_documents(
        docs
    )
    print("Extracting content material with LLM")

    # Seize the primary 1000 tokens of the positioning
    splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
        chunk_size=1000, chunk_overlap=0
    )
    splits = splitter.split_documents(docs_transformed)

    # Course of the primary break up
    extracted_content = extract(schema=schema, content material=splits[0].page_content)
    pprint.pprint(extracted_content)
    return extracted_content

# load from the online and scrape the dataset
urls = ["https://www.wsj.com"]
extracted_content = scrape_with_playwright(urls, schema=schema)

In above code, we’ve created a perform known as “scrape_with_playwright” to load and remodel net web page information from any web site or sequence of internet sites with using up to date schema of the the output content material in title and abstract format as proven above within the code instance.

Query Answering Over a Internet

Now, Creating Q&A software for automating net analysis utilizing queries will be archived by way of Google Search APIs and strategies like ‘WebResearchRetriever’ from LangChain framework. To start with the applying growth first we’ll look the the applying workflow diagram to know key elements of the such LLM primarily based software.

The diagram above illustrates your complete course of from a analysis query to net scraping and net content material storage. It additionally exhibits how LLM APIs are known as to generate complete responses to person queries utilizing the scraped net content material and context. Such an software can scale back our reliance on conventional search strategies. To construct software, first we have to set up sure libraries within the undertaking surroundings as talked about under code.

# requirement.txt
langchain==0.2.5
langchain-chroma==0.1.1
langchain-community==0.2.5
langchain-core==0.2.9
langchain-openai==0.1.9
chromadb==0.5.3
openai==1.35.3 
html2text==2024.2.26
google-api-core==2.11.1
google-api-python-client==2.84.0
google-auth==2.27.0
google-auth-httplib2==0.1.1
googleapis-common-protos==1.63.1
tiktoken==0.7.0

As soon as the set up is full import the required libraries and set OpenAI API keys in addition to Google API keys.

# importing langchain instruments
from langchain.retrievers.web_research import WebResearchRetriever
from langchain_chroma import Chroma
from langchain_community.utilities import GoogleSearchAPIWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

# importing os and organising 
import os
os.environ["GOOGLE_API_KEY"] = "YOUR_API_KEY"
os.environ["GOOGLE_CSE_ID"] = "YOUR_CSE_ID" 
os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY"

For extra particulars on find out how to set above talked about API keys. Go to this hyperlink for additional steerage. Subsequent, we’ll initialize the vector retailer, LLM, and Google search cases. Following this, we’ll arrange the online retrieval device to make use of the LLM for producing a number of queries. These queries will then be executed on Google search APIs to carry related net hyperlinks. The retrieved net pages can be scraped and loaded into vector shops, which can function context for reply technology.

# Vectorstore storage
vectorstore = Chroma(
    embedding_function=OpenAIEmbeddings(), persist_directory="./chroma_db_oai"
)

# LLM occasion
llm = ChatOpenAI(temperature=0)

# Search API occasion
search = GoogleSearchAPIWrapper()

# Initialize the online retrival 
web_research_retriever = WebResearchRetriever.from_llm(
    vectorstore=vectorstore, llm=llm, search=search
)

As soon as the online retriever is about, now we simply have to enter person question to utilizing Q&A retrieval chain of langchain for technology of reply from huge array of net assets.

# Run the q&a retrival chain
import logging

logging.basicConfig()
logging.getLogger("langchain.retrievers.web_research").setLevel(logging.INFO)
from langchain.chains import RetrievalQAWithSourcesChain

# take a person enter and use q&a series for net retrival
user_input = "How do LLM Powered Autonomous Brokers work?"
qa_chain = RetrievalQAWithSourcesChain.from_chain_type(
    llm, retriever=web_research_retriever
)

# print the lead to your surroundings 
outcome = qa_chain({"query": user_input})

From the code above, it’s evident that by using the person question alongside the retrieval chain. We are able to generate a complete and summarized reply from an enormous array of net pages.

Conclusion

On this article, we’ve explored the creation of an online automation software leveraging LangChain and Google Search APIs. We began with an introduction to the online automation workflow, outlining the steps concerned in remodeling uncooked net information into precious info for a given person question. We then delved into the specifics of net loading and information transformation, important for making ready the information for additional processing.

Following this, we mentioned find out how to carry out scraping and extraction utilizing LangChain. Additionally highlighting its capabilities in effectively gathering and processing net information. Lastly, we demonstrated find out how to implement a question-answering system over the online for analysis functions. This technique offers fast and complete solutions from net assets with out the necessity to undergo each individually.

Key Takeaways

• This text affords a hands-on information to creating net automation functions. Additionally demonstrating sensible use-cases and advantages of integrating AI-powered instruments in search processes.
• Understanding the whole workflow, from net loading and information transformation to scraping and query answering for net automation software.
• Leveraging LangChain and Google Search APIs considerably improves search effectivity by offering succinct, summarized solutions and linking to related assets.

Steadily Requested Questions

The media proven on this article isn’t owned by Analytics Vidhya and is used on the Writer’s discretion.