You scrape Google search results by using a program, or a paid service that runs one, to collect the titles, links and other parts of the results page into a file or database. People do it to track rankings, study competitors or feed data into tools. There are several ways to get that data, and they differ a lot in cost, reliability and how they sit with Google’s rules. This page explains what Google says first, then the main methods, then where our own tool fits. For tools built only for the questions box, see our comparison of the best People Also Ask tools.
One note before we start: this is a plain-English summary of published rules, not legal advice. Laws on scraping differ by country, and a company’s terms are not the same thing as the law.
What do Google’s terms say about scraping?
Google covers automated access in two places: its Terms of Service, which apply to all its services, and its spam policies for Search.
Google Terms of Service
The current Google Terms of Service (effective 30 July 2026) list things you “must not” do under the heading “Don’t abuse our services”. Three of them bear on scraping:
- “using automated means to access content from any of our services in violation of the machine-readable instructions on our web pages (for example, robots.txt files that disallow crawling, training, or other activities)”
- “spamming, hacking, or bypassing our systems or protective measures”
- “hiding or misrepresenting who you are in order to violate these terms”
The same terms say Google may suspend or end your access if your conduct causes harm, giving “scraping content that doesn’t belong to you” as one example.
Google’s robots.txt
The first rule points at robots.txt, so it matters what Google’s own file says. When we checked google.com/robots.txt on 5 November 2026, the rules for all user agents began with Disallow: /search. In other words, Google’s machine-readable instructions ask automated crawlers to stay out of its search results pages.
Google’s spam policies
The spam policies for Google web search (last updated 28 August 2026) are more direct. Under “Machine-generated traffic” they say it “refers to the practice of sending automated queries to Google. This includes scraping results for rank-checking purposes or other types of automated access to Google Search conducted without express permission.” And: “Such activities violate our spam policies and the Google Terms of Service.”
The phrase that matters is “without express permission”. Google’s position is that automated searching needs its permission, whatever tool you use to do it.
Google v. SerpApi: the rules in court
On 19 December 2025, Google sued SerpApi, a company that sells search results data through an API. Google’s general counsel wrote that the suit was filed against SerpApi “for circumventing security measures protecting others’ copyrighted content that appears in Google search results”, and that Google devotes “significant resources to fighting this abuse” (Google). Search Engine Roundtable reported that Google argues its security systems, such as SearchGuard, are technological measures that control access to copyrighted work, so bypassing them would breach Section 1201 of the DMCA (Search Engine Roundtable).
SerpApi disputes this. In a post on 25 August 2026, it said the court had dismissed Google’s original complaint in July, that Google had filed a narrower amended complaint, and that SerpApi had asked the court to dismiss that one too. It describes its service like this: “SerpApi provides programmatic access to search results that are publicly visible to anyone using a standard web browser” (SerpApi).
We don’t take a side on the case, and it was still open in SerpApi’s latest post. The practical point is simpler: Google treats large-scale scraping as something to fight, technically and in court, so any method that depends on it carries risk for whoever runs it.
The main ways to get Google results data
Here is how the common options compare, as of 5 November 2026:
| Method | What you get | Good for | Watch out for |
|---|---|---|---|
| Google Custom Search JSON API | Results from a Programmable Search Engine you set up, as JSON | Existing customers only | Closed to new customers; ends 1 January 2027 |
| SERP API providers | Structured results pages for keywords and locations you choose | Rank tracking and research at scale | Cost, and how the provider gets its data |
| Search Console export | Queries, clicks, impressions and positions for your own site | Your own performance | Only your site, no competitors |
| Manual copy or export | Whatever you can see on the page | A handful of queries | Slow; one sample in time and place |
| Writing your own scraper | Raw results pages | Not recommended | Against Google’s terms and spam policies; blocked by Google’s defenses |
Google’s Custom Search JSON API
For years this was the official answer: an API that returns results from a Programmable Search Engine in JSON. It is now on its way out. Google’s developer page (last updated 18 February 2026) says: “The Custom Search JSON API is closed to new customers.” It adds: “Existing Custom Search JSON API customers have until January 1, 2027 to transition to an alternative solution.”
Until then, existing customers get 100 free queries a day, and Google lists extra requests at $5 per 1,000 queries, up to 10,000 a day. For new projects Google points elsewhere: it calls Vertex AI Search “a favorable alternative for searching up to 50 domains”, and for full web search it asks you to contact Google about its full web search solution. The page doesn’t publish a price for that, so see Google’s page for details.
Note what the API never was: a copy of the normal Google results page. It searched an engine you configured, so its results and features did not have to match what a searcher sees on google.com.
SERP APIs
A SERP API is a paid service that takes a keyword, location and device, and returns the results page as structured data: organic results, ads, People Also Ask, local packs and so on. Two examples, from their own pages as of 5 November 2026:
- DataForSEO describes its SERP API with pay-as-you-go pricing and “Turnaround time of up to 6 seconds”, and says you can get data via n8n, Make, ChatGPT or Claude.
- SerpApi describes itself as programmatic access to publicly visible search results, and is the company Google is suing.
For prices, see each provider’s pricing page. Before you sign up, read the provider’s terms and ask how it collects results, because Google’s spam policy applies to automated access “conducted without express permission” whoever does the collecting. If a provider can’t explain its approach clearly, that tells you something.
Search Console export
If what you want is your own site’s data, you don’t need to scrape anything. Google Search Console’s Performance report shows the queries, clicks, impressions and average position for your site, and Google says “Many reports provide an export button to download both the chart and table data” (Search Console Help). It is free, it comes from Google, and it shows real searches rather than one test search from one place. It won’t show you competitors, which is why people look elsewhere.
Manual copy and export
For a few queries, searching by hand and copying what you see into a sheet is often enough. Record the date, country and device, because results vary with all three. It doesn’t scale, but for checking a handful of keywords before you write a page it is free and needs no special tools.
Is there a free Google scraper?
Free “Google scraper” scripts and open-source projects are easy to find. They don’t change anything above. Running one is still automated access to Google Search, which the spam policies say needs express permission, and Google says it devotes significant resources to stopping this kind of traffic. Expect blocks and CAPTCHAs, and expect them to keep changing. We won’t give advice on getting around Google’s protections: the Terms of Service list “bypassing our systems or protective measures” as abuse, and that advice ages badly anyway.
If you only need a small, specific part of the page, it is usually easier to find a tool built for that part.
Where our tool fits
PAA Box is our free Chrome extension, and it covers one small part of the page: the People Also Ask box. It is not a Google scraper or a rank tracker. It works in your own Chrome, on the results page you have open, and opens the People Also Ask questions up to four levels deep (D0 to D3). You can then export a CSV with each question’s depth and parent question.
It does not read the organic results, ads, the AI Overview, AI Mode, People Also Search For, related searches or autocomplete. If you need those, one of the methods above is the right tool. If you only need the questions, see how to export People Also Ask questions, or how to bulk extract People Also Ask questions across many searches.
If you work in an AI assistant, our MCP connector lets apps like Claude run those same unpacks for you, still inside your own Chrome.
Which method should you choose?
- Your own site’s rankings and clicks: Search Console. It is free and comes from Google.
- A few keywords, once: search by hand and keep a dated note.
- Ongoing rank tracking for many keywords: a rank tracker or SERP API whose terms and data collection you have read and accept.
- The questions people ask around a topic: People Also Ask, collected by hand or with a People Also Ask tool.
- A new app that needs web search: not the Custom Search JSON API, which is closed to new customers and will be discontinued on 1 January 2027. Look at the alternatives Google names on its developer page.
Whatever you choose, keep a record of where the data came from and when. Results change by time, place and device, so a dated, sourced sample is worth more than a big file nobody can explain.