sherpa_ai.scrape package

Overview

The scrape package provides utilities for extracting and processing information from external sources like files, websites, and repositories. These tools enable agents to gather relevant information from structured and unstructured data sources.

Key Components

• FileScaper: Tools for extracting content from local files

• GitHubReadmeExtractor: Utilities for retrieving README content from GitHub

Example Usage

from sherpa_ai.scrape.file_scraper import FileScraper
from sherpa_ai.scrape.extract_github_readme import GitHubReadmeExtractor

# Extract content from local files
scraper = FileScraper()
content = scraper.scrape_file("path/to/document.txt")

# Parse Python files for structured content
python_content = scraper.scrape_python_file("path/to/script.py")
print(f"Found {len(python_content['classes'])} classes")
print(f"Found {len(python_content['functions'])} functions")

# Extract README from a GitHub repository
github_extractor = GitHubReadmeExtractor()
readme = github_extractor.extract_readme("username", "repository")
print(readme)

Submodules

Module	Description
sherpa_ai.scrape.extract_github_readme	Utilities for retrieving and processing README files from GitHub repositories.
sherpa_ai.scrape.file_scraper	Tools for extracting and parsing content from local files in various formats.

sherpa_ai.scrape.extract_github_readme module

GitHub README extraction module for Sherpa AI.

This module provides functionality for extracting and processing README files from GitHub repositories. It handles authentication and content extraction.

sherpa_ai.scrape.extract_github_readme.get_owner_and_repo(url)

Extract owner and repository name from GitHub URL.

This function parses a GitHub repository URL to extract the owner’s username and repository name.

Parameters:

url (str) – GitHub repository URL (e.g., ‘owner/repo’).

Returns:

A tuple containing:

• owner (str): Repository owner’s username

• repo (str): Repository name

Return type:

tuple[str, str]

Example

>>> url = "https://github.com/openai/gpt-3"
>>> owner, repo = get_owner_and_repo(url)
>>> print(owner, repo)
'openai' 'gpt-3'

sherpa_ai.scrape.extract_github_readme.extract_github_readme(repo_url)

Extract README content from a GitHub repository.

This function downloads and extracts the content of a repository’s README file (either .md or .rst).

Parameters:

repo_url (str) – GitHub repository URL.

Returns:

README content if found and successfully extracted,

None otherwise.

Return type:

Optional[str]

Example

>>> url = "https://github.com/openai/gpt-3"
>>> content = extract_github_readme(url)
>>> if content:
...     print(content[:50])
'# GPT-3: Language Models are Few-Shot Learners...'

sherpa_ai.scrape.file_scraper module

File scraping and handling module for Sherpa AI.

This module provides functionality for downloading, processing, and analyzing files attached to questions. It handles various file types including PDF, text, markdown, HTML, and XML files.

class sherpa_ai.scrape.file_scraper.QuestionWithFileHandler(question, files, token, user_id, team_id, llm)

Bases: object

Handler for questions with attached files.

This class manages the process of downloading, processing, and analyzing files attached to questions. It supports various file types and handles token limits and content summarization.

question

The user’s question to be answered.

Type:

str

token

OAuth token for file access.

Type:

str

files

List of file information dictionaries.

Type:

list

user_id

ID of the user asking the question.

Type:

str

llm

Language model for text processing.

Type:

Any

Example

>>> handler = QuestionWithFileHandler(
...     question="What's in the document?",
...     files=[{"id": "123", "filetype": "pdf"}],
...     token="oauth_token",
...     user_id="user123",
...     llm=language_model
... )
>>> result = handler.reconstruct_prompt_with_file()
>>> print(result["status"])
'success'

reconstruct_prompt_with_file()

Reconstruct the prompt using the attached file.

This method downloads the file, processes its content, and combines it with the original question to create a more informed prompt.

Returns:

A dictionary containing:

• status (str): ‘success’ or ‘error’

• data (str): Reconstructed prompt if successful

• message (str): Error message if failed

Return type:

dict

Example

>>> result = handler.reconstruct_prompt_with_file()
>>> if result["status"] == "success":
...     print(result["data"])
'Based on the PDF content...'

download_file(file)

Download and extract content from a file.

This method downloads a file using its URL and extracts its content based on the file type. Supports PDF, text, markdown, HTML, and XML.

Parameters:

file (dict) – File information dictionary containing: - id (str): File identifier - mimetype (str): MIME type - url_private_download (str): Download URL - filetype (str): File extension

Returns:

A dictionary containing:

• status (str): ‘success’ or ‘error’

• data (str): File content if successful

• message (str): Error message if failed

Return type:

dict

Example

>>> file_info = {
...     "id": "123",
...     "filetype": "pdf",
...     "url_private_download": "https://example.com/doc.pdf"
... }
>>> result = handler.download_file(file_info)
>>> if result["status"] == "success":
...     print(len(result["data"]))
1024

prompt_reconstruct(file_info, data=<class 'str'>)

Reconstruct the prompt with file content.

This method processes the file content, handles token limits, and combines the content with the original question to create an enhanced prompt.

Parameters:

• file_info (dict) – File information dictionary containing: - filetype (str): File extension - name (str): File name - title (str): File title

• data (str) – Content of the file.

Returns:

A dictionary containing:

• status (str): ‘success’ or ‘error’

• data (str): Reconstructed prompt if successful

• message (str): Error message if failed

Return type:

dict

Example

>>> file_info = {
...     "filetype": "pdf",
...     "name": "document.pdf",
...     "title": "Important Doc"
... }
>>> result = handler.prompt_reconstruct(file_info, "content...")
>>> if result["status"] == "success":
...     print(result["data"])
'Based on the PDF "Important Doc"...'

Module contents