Save extracted URL's in Wayback Machine

[finnless]

URL's should be preserved as soon as they are found.

Reference code:

pragma.archivelab.org/pragma/api/pragmas.py
Lines 51 to 72 in ArchiveLabs/pragma.archivelab.org@7587f6d

def save(url):
    url = url if '://' in url else 'http://' + url
    r = requests.get('http://web.archive.org/save/%s' % url)    
    if 'X-Archive-Wayback-Runtime-Error' in r.headers:
        return {
            'error': r.headers['X-Archive-Wayback-Runtime-Error']
        }
    print(r.headers)
    content_location = r.headers.get('content-location', url)
    if 'x-archive-wayback-liveweb-error' in r.headers:
        raise core.HTTPException(r.headers['x-archive-wayback-liveweb-error'],
                                 r.status_code)
    protocol = 'https' if 'https://' in content_location else 'http'
    uri = content_location.split("://")[1] 
    path = uri[uri.index('/'):] if uri.index('/') is not None else '/';
    return {
        'date': r.headers['date'],
        'protocol': protocol,
        'domain': uri.split('/')[0],
        'path': path,
        'id': content_location
    }

Scope

• Daily cron that runs @jjjake's derive module to generate genomes for books, for the resulting genome.json, test curl any urls, and then archive in wayback. This is also related to Add copyright page detector module and extractor #51 and and Add Open Library Tables of Contents → Solr internetarchive/openlibrary#8756 as the same job should/could likely also handle TOC identification and extraction.

[mekarpeles]

• Open Library has ~1M books containing URLS: https://openlibrary.org/search/inside?q=http%3A%2F%2F&mode=everything
  We have a project called the Open bookgenomeproject.org which uses a sequencer bot to read books and produce "book_genome" files with insights
• Here are ~13k books which have book_genomes: https://archive.org/search.php?query=format%3Abook_genome
• For one of these items having ID isbn_9791570594938, you can view its genome by going to: https://archive.org/download/isbn_9791570594938/book_genome.json
• Within the book_genome.json, we can see 2 urls were discovered in the book above
• The IA tool can be used to download book_genome.json files for items: https://archive.org/services/docs/api/internetarchive/internetarchive.html
• There is a feature calls Save Page Now which can be used to save these urls in the Wayback Machine: https://blog.archive.org/2019/10/23/the-wayback-machines-save-page-now-is-new-and-improved
• I've also created an API which one can programatically use to accomplish this... https://github.com/ArchiveLabs/pragma.archivelab.org/blob/master/pragma/api/pragmas.py#L51-L72

The hard part is discovering URLs, but @finnless has a dump of all the book_genome.json files and can answer questions about figuring out which ones contain urls!

[mekarpeles]

Saving in the Wayback Machine (SPN2)

Since writing the gitub issue, there is now a Save Page Now 2 (SPN2) API you can use to save urls in the wayback machine.

Here is a colab demo which you can use with your account's s3 keys: https://colab.research.google.com/drive/1y7c4n34N6oahlMjImDfG1QzembHWUJW4#scrollTo=M4shYxuf_AFj

You can also see this python example of SPN2 here:
internetarchive/openlibrary#10577

How to approach this challenge

There are two options:

1. Pull down the fulltext of each book and identify URLs OR
2. Start with the gnome data that already exists...

And in both cases: check + fix these URLs (using string matching, regex, or AI/LLMs, etc), save them in the wayback machine, keep track of which urls were saved per book & page.

[hunter-nguyen]

Hi @mekarpeles !

I just saw your recent update about the SPN2 API and the Colab demo! It's great to see such recent advancements, and I'm really interested in contributing to this project and would love to dive into improving the URL extraction process and integrating it more seamlessly with the Wayback Machine. I'm currently exploring the book_genome.json files and looking at how we can extract and clean URLs before saving them.

I’ve also been looking into the SPN2 API and wanted to ask if there are any specific areas of the process where you'd recommend I start—perhaps testing the API or focusing on specific aspects of URL detection? I’d be happy to help in any way I can!

Hunter

[Prerna1313]

Hi everyone!

I’d love to contribute to this issue by developing a system to extract and archive URLs from book_genome.json files using the Save Page Now 2 (SPN2) API.

Proposed Approach:
🔹 Step 1: Extract URLs from book_genome.json files (via the Internet Archive API).
🔹 Step 2: Validate and clean URLs (using regex, AI-based parsing, etc.).
🔹 Step 3: Archive URLs using the Save Page Now 2 (SPN2) API.
🔹 Step 4: Keep track of archived URLs per book & page.

Implementation Considerations:
✔ SPN2 API Integration: Using the [SPN2 Python example] (internetarchive/openlibrary#10577).
✔ Colab Testing: Will explore the SPN2 Colab demo to test saving URLs.
✔ Approach Selection: Should I start directly with book_genome.json data, or pull full-text data for URL extraction preferred?
✔ Tracking & Logging: Storing saved URLs per book/page.

Are there any formatting preferences for storing archived URLs? Should failed archiving attempts be logged separately? Any specific test cases or edge cases to consider?

Looking forward to feedback before I begin coding!

Thanks
Prerna Kashyap

[finnless]

Hello @hunter-nguyen and @Prerna1313 ! Both of your proposals for beginning to explore implementing this feature sound great! Don't feel like you need permission to start experimenting with this. You are welcome to share your progress in this issue or open a new PR with a draft implementation. Thank you both for your interest!

I'd recommend using the urls extracted in the existing pipeline and adding a step that tests these urls and sends the ones that are live to SPN2 as indicated in the issue's Scope.

[CloudsProgram]

Hi @finnless , Is there discord or slack I can join to inquire more about this specific project? Or should I just post it here?

[gsreeja11]

Hi @finnless , I have filled the form to be a contributor and yet have not received a link to the slack channel :( I very much want to attend the community calls and collaborate. And I am super eager to be a GSoC contributor for this project.

Also I would love to have your thoughts of my current idea to implement the URLs present in Books -

Regarding Open Genome , here is my Draft implementation + Obervations.

Right now, the sample code given in colab runs a bit slow as it does not use multiple threads, 100 takes a bit of time - I have optimised to the below

import requests
from collections import defaultdict
from concurrent.futures import ThreadPoolExecutor, as_completed

api_url = "https://archive.org/advancedsearch.php"
MAX_THREADS = 20  
ROWS = 2000       

bookToURLMapping = defaultdict(list)


def get_genome_identifiers(rows=ROWS):
    query = "format%3Abook_genome"
    page = 1
    params = {
        'query': query,
        'rows': rows,
        'api_url': api_url,
        'page': page
    }
    url = "{api_url}?q={query}&fl%5B%5D=identifier&page={page}&rows={rows}&output=json".format(**params)
    r = requests.get(url)
    r.raise_for_status()
    return [doc['identifier'] for doc in r.json()['response']['docs']]

def fetch_genome(identifier):
    try:
        print(f"Fetching: {identifier}")
        item_dl_url = f"https://archive.org/download/{identifier}/book_genome.json"
        r = requests.get(item_dl_url)
        r.raise_for_status()
        urlsInBook = r.json().get('urls', [])
        bookToURLMapping[identifier].append(urlsInBook)
        bookToURLMapping[identifier].append([len(urlsInBook)])
        return urlsInBook
    except Exception as e:
        print(f"Error fetching book_genome for {identifier}: {e}")
        return []

def fetch_all_genomes(identifiers):
    list_of_urls = []
    with ThreadPoolExecutor(max_workers=MAX_THREADS) as executor:
        future_to_id = {executor.submit(fetch_genome, id_): id_ for id_ in identifiers}
        for future in as_completed(future_to_id):
            urls = future.result()
            list_of_urls.append(urls)
    return list_of_urls

identifiers = get_genome_identifiers()
listOfUrls = fetch_all_genomes(identifiers)

print(f"\nFetched Genome URLs for {len(listOfUrls)} books.")

Here I have processed 2000 books having genomes, extracted URLs and saved in a dictionary. I know logging and other aspects are missing but this is very much a draft right now

This is the draft URL validator I am using (very simple and in progress)

import re
import requests
from urllib.parse import urlparse, urlunparse

def clean_url(raw_url):

    url = raw_url.strip()


    if url.startswith("ttp://"):
        url = "h" + url
    elif url.startswith("ttps://"):
        url = "h" + url
    elif not url.startswith(("http://", "https://")):
        url = "http://" + url  # fallback


    url = url.replace(" ", "")
    url = url.replace("\n", "")


    url = re.sub(r"[)\].,;!?]+$", "", url)

    return url

def is_valid_url(url):
 
    url_regex = re.compile(
        r"^http?://"
        r"(([A-Za-z0-9-]+\.)+[A-Za-z]{2,})" 
        r"(:\d+)?(/[\w\-./?%&=]*)?$"
    )
    return re.match(url_regex, url) is not None

def is_url_accessible(url, timeout=5):

    try:
        response = requests.head(url, timeout=timeout, allow_redirects=True)
        if response.status_code < 400:
            return True, response.status_code
        # Try GET if HEAD fails
        response = requests.get(url, timeout=timeout, allow_redirects=True)
        return response.status_code < 400, response.status_code
    except Exception as e:
        return False, str(e)

I am assuming all 200s as OK for now. Also, in-progress - Using LLMs to fix broken URLs I seem to be having some success here , will udpdate

And finally, saving using SPNs

import time
for book_id, value in bookToURLMapping.items():
  print('Processing book', book_id) 
  listOfUrls = value[0]
  for url in listOfUrls:
    print('Processing url', url) 
    if url:
      url = clean_url(url)
      if is_valid_url(url) and is_url_accessible(url):
        print('Clean url', url) 
        time.sleep(5)
        data = {
        "url": url,
        "if_not_archived_within": "1d",
        "capture_all": "1"
      }
      response = requests.post(api_url, headers=headers, data=data)
      print(response.json())

Some questions

1. As the deadline to submit the proposal is tomorrow, could I pelase request a review of my document. I have filled the form already mentioned saying a mentor would be assigned. Or could I send over the document please to your email
2. Please share the slack channel link
3. I am working on refining the URL validator right now, and working on incorporating LLMs/AI to fix URL and check links, is that fine
4. While saving using SPN, some URLs like

Processing url https://archive.org/details/environmentalcauooooeedy
Clean url https://archive.org/details/environmentalcauooooeedy
{'message': 'We’re currently facing some limitations when it comes to archiving this site. We apologize for any inconvenience this might cause and appreciate your understanding. Please email us at "[info@archive.org](mailto:info@archive.org)" if you would like to discuss this more.', 'status': 'error', 'status_ext': 'error:blocked-url'}

Seem to be blocked, and unable to be saved.

5. Other URLs are valid but it seems that someone deleted it from internet archive, what to do in those cases
6. In cases where everything works and URL is saved,

Clean url http://www.shrm.org/
{'url': 'http://www.shrm.org/', 'job_id': 'spn2-cf45e4027bc957a50b936f22a646c3f9908c63ff'}

I am working on adding telemetry like how many were saved etc., do you think a DB storage layer makes sense ?

7. I have exceeded max retries to /save endpoint, any way to bypass that please
   ConnectionError: HTTPSConnectionPool(host='web.archive.org', port=443): Max retries exceeded with url: /save (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f6fc3d51a90>: Failed to establish a new connection: [Errno 111] Connection refused'))

Sorry for so many questions at once, just super eager to get started on this project :)

[anas1bk]

hello @finnless i hope you're doing well, i have a few questions, first one would be ; is there any platform that would facilitate communication with mentors like discord or slack, other questions are as followed;

1. 📘 Book Genomes
   I’ve already built a pipeline that processes OCR and archives URLs to the Wayback Machine. I’m currently integrating Book Genome processing to extract URLs from book_genome.json files. Is there an official or preferred schema for aligning genome-extracted URLs with specific pages or page regions to help with BookReader integration?
2. 🔵 BookReader Integration
   Once URLs are extracted and archived, is there a reference implementation or plugin template for injecting the clickable region overlays into BookReader based on coordinates? Is the https://api.archivelab.org/books/{identifier}/pages/{page}/ocr?mode=words endpoint the best source for aligning text positions with link overlays?
3. 🧠 Error Recovery / Archival Failures
   In some cases, I receive blocked-url or transient 523/503 errors from the Wayback Save Page Now API. Is there a list of known blocked domains? Should these be logged and skipped, or do you recommend retries or whitelisting requests?
4. 📈 Scaling + Dataset
   I’ve been running tests on a few hundred books successfully. Are there recommended methods or formats for submitting data back to Internet Archive once a batch of URLs is archived per book (e.g., urls.json, BookReader metadata, or S3 upload paths)?
5. 🤝 Community Collaboration
   Would it be helpful if I published some of the structured URL data I’ve collected (including OCR sources, archival status, and timestamps) for other contributors to validate or build upon? If yes, what format would be preferred for community sharing?
6. 🧪 OBGP Integration
   In terms of long-term contribution, is there value in enriching OBGP genome files with archival metadata (e.g., archived: true, or wayback_url) so future contributors can skip rechecking?
   if anyone would like to discuss further more, mentor or contributer here's my discord 😁 : anass._.bkk