arXiv Papers Search Scraper

arXiv Papers Search Scraper is a BrowserAct Bot template for collecting structured research paper records from public arXiv search results. Configure a keyword, search field, and result count, then run the Bot to return ranked paper records for literature review, academic monitoring, AI research tracking, data enrichment, or automation workflows.

What Does BrowserAct arXiv Papers Search Scraper Do?

This template opens arXiv search for your keyword, reads public paper result cards, follows result pagination when needed, and returns one structured record per paper.

Key Features

Extract public arXiv search results without requiring a login.
Capture rank, title, arXiv URL, arXiv ID, authors, abstract, categories, submitted date, comments, journal reference, PDF URL, and keyword.
Configure base URL, keyword, search type, and count.
Continue through arXiv pagination until the requested count is reached.
Keep paper IDs and PDF links ready for citation databases, research trackers, and agent workflows.

What Data Can I Extract from arXiv?

Field	Description
`rank`	Search result position.
`title`	Paper title.
`arxiv_url`	Public arXiv abstract page URL.
`arxiv_id`	arXiv paper identifier.
`authors`	Paper authors shown in the result.
`abstract`	Abstract snippet from the search listing.
`categories`	arXiv subject categories.
`submitted`	Submitted or announced date text.
`comment`	Comment text when visible.
`journal_ref`	Journal reference when visible.
`pdf_url`	Direct public PDF URL.
`search_keyword`	Keyword used for the run.

How to Use arXiv Papers Search Scraper

Click Run task.
Enter a keyword or research topic.
Choose the arXiv search type, such as all, title, author, abstract, or full_text.
Set how many paper records to collect.
Run the Bot and review the structured paper results.
Reuse the Bot with another topic or connect the output to your workflow.

Input Parameters

Parameter	Required	Default	Description
`base_url`	Yes	`https://arxiv.org`	arXiv base URL without a trailing slash.
`keyword`	Yes	`large language model`	Search query text.
`search_type`	Yes	`all`	arXiv search field, such as `all`, `title`, `author`, `abstract`, `comments`, `journal_ref`, `paper_id`, `doi`, or `full_text`.
`count`	Yes	`20`	Maximum number of paper records to return.

Example Output

{
  "results": [
    {
      "rank": 1,
      "title": "SPADE: Self-Play in Adaptive Synthetic Executable Environments",
      "arxiv_url": "https://arxiv.org/abs/2608.19197",
      "arxiv_id": "2608.19197",
      "authors": "Bo Liu, Simon Yu, Yiding Jiang",
      "abstract": "Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals.",
      "categories": "cs.CL; cs.AI",
      "submitted": "Submitted 19 August, 2026; originally announced August 2026.",
      "comment": "Work in progress.",
      "journal_ref": "",
      "pdf_url": "https://arxiv.org/pdf/2608.19197",
      "search_keyword": "large language model"
    }
  ]
}

How Does it Work?

How arXiv Papers Search Scraper runs

BrowserAct constructs a public arXiv search URL from base_url, keyword, and search_type, opens the result page, and reads each paper listing. The Bot extracts the visible arXiv abstract URL, paper ID, title, authors, abstract snippet, categories, submitted date, comments, journal reference, and PDF link.

When count is higher than the first page, BrowserAct follows arXiv result pagination using the public start offset until the requested number of paper records is collected or no more results are available. Optional fields such as journal_ref may be empty when arXiv does not show them for a paper.

How to Build a New Scraper Bot with Agent Built

Step 1: Describe the Data You Need

Open BrowserAct and start Agent Built from the Home prompt box or from Create -> Build with Agent. Paste a clear request that includes the website, the records to collect, the filters or search conditions, the fields to return, the result limit, and the inputs you want to reuse later.

Agent Built prompt templates

Step 2: Let BrowserAct Build and Test the Bot

BrowserAct explores the live website, works out the navigation and extraction path, and validates the result. If BrowserAct asks for clarification, reply in the same build conversation with the missing URL, condition, field, example, or expected output.

Agent Built exploration

Step 3: Run the Bot and Review the Result

When the build is complete, run the Bot with the default inputs first. Review the structured records, source URLs, missing fields, and row count before reusing the Bot with new inputs or connecting it to another workflow.

Agent Built run inputs

Agent Built run results

Why Scrape arXiv Paper Search Results?

arXiv search data helps teams monitor emerging research, collect literature-review candidates, track fast-moving AI topics, enrich paper databases, and route academic records into summaries, dashboards, alerts, or agent workflows.

Who Can Use This Template?

Researchers collecting papers for literature reviews.
AI teams tracking new model, benchmark, or agent papers.
Developer relations teams monitoring technical research trends.
Data teams enriching internal knowledge bases with public paper metadata.
Automation builders routing paper records into spreadsheets, databases, alerts, or agents.

How Many Results Can You Scrape?

Use count to control the number of paper records returned. Higher counts require moving through more arXiv result pages and may take longer depending on arXiv response speed.

Automation and Export

Automation and export flow

BrowserAct returns structured arXiv paper records first. You can then use the output through API JSON responses, n8n workflows, Make scenarios, Zapier automations, MCP-based agent workflows configured from the Bot's Integrations tab, or manual CSV review.

Need Help?

Discord: Discord Community

E-mail: service@browseract.com

Brief