Scraping SERPs for Archival Seeds: It Matters When You Start

by   Alexander C. Nwala, et al.

Event-based collections are often started with a web search, but the search results you find on Day 1 may not be the same as those you find on Day 7. In this paper, we consider collections that originate from extracting URIs (Uniform Resource Identifiers) from Search Engine Result Pages (SERPs). Specifically, we seek to provide insight about the retrievability of URIs of news stories found on Google, and to answer two main questions: first, can one "refind" the same URI of a news story (for the same query) from Google after a given time? Second, what is the probability of finding a story on Google over a given period of time? To answer these questions, we issued seven queries to Google every day for over seven months (2017-05-25 to 2018-01-12) and collected links from the first five SERPs to generate seven collections for each query. The queries represent public interest stories: "healthcare bill," "manchester bombing," "london terrorism," "trump russia," "travel ban," "hurricane harvey," and "hurricane irma." We tracked each URI in all collections over time to estimate the discoverability of URIs from the first five SERPs. Our results showed that the daily average rate at which stories were replaced on the default Google SERP ranged from 0.21 -0.54, and a weekly rate of 0.39 - 0.79, suggesting the fast replacement of older stories by newer stories. The probability of finding the same URI of a news story after one day from the initial appearance on the SERP ranged from 0.34 - 0.44. After a week, the probability of finding the same news stories diminishes rapidly to 0.01 - 0.11. Our findings suggest that due to the difficulty in retrieving the URIs of news stories from Google, collection building that originates from search engines should begin as soon as possible in order to capture the first stages of events, and should persist in order to capture the evolution of the events...


page 7

page 8

page 9


SHARI – An Integration of Tools to Visualize the Story of the Day

Tools such as Google News and Flipboard exist to convey daily news, but ...

Measuring News Similarity Across Ten U.S. News Sites

News websites make editorial decisions about what stories to include on ...

Designing and Evaluating Interfaces that Highlight News Coverage Diversity Using Discord Questions

Modern news aggregators do the hard work of organizing a large news stre...

Stories From the Past Web

Archiving Web pages into themed collections is a method for ensuring the...

Novelty in news search: a longitudinal study of the 2020 US elections

The 2020 US elections news coverage was extensive, with new pieces of in...

Who Provides the Largest Megaphone? The Role of Google News in Promoting Russian State-Affiliated News Sources

The Internet has not only digitized but also democratized information ac...

Automated Query Generation for Evidence Collection from Web Search Engines

It is widely accepted that so-called facts can be checked by searching f...

Please sign up or login with your details

Forgot password? Click here to reset