I am trying to load files that are between 273 MB and 2 GB. I am using a Mac. However, the files stall halfway through when loading. I do not have this issue with small files. These are WARC files that were created using Conifer.

I'm not sure if this behaviour is by design. I'm often downloading WARCs so I can consume them later on when I'm without internet connection.

I installed replayweb.page as a PWA (Chrome) and whenever I open previously added archive, it starts loading I assume live version of the archive. It takes many seconds to show me the page, why is that so? I would expect it to be instant (or maybe show local copy first and then try to re-fetch).

Same thing happens when I open PWA app with flaky connection - it hangs on splash screen. It's quite annoying, unfortunately PWA does not have any settings for just using local archives.

subj

A slight difference with WR Player is that the markdown rendering is not quite the same, as mention in this thread:
https://forum.webrecorder.net/t/using-replayweb-page-app-story-pka-collection-overview/43/3

The spacing should ideally be comparable.

Sorry if this is the wrong place to ask this type of question but i didn't know where else to go.

Basically I want to hide the website address, local address, client area (file, edit, view etc).

Is this possible to do? Basically I use this app to teach english from slides I downloaded off a website but the top part makes it too clunky.

Another quick thing I noticed, but am not (at least yet) addressing in the aforementioned accessibility PR...

The regex used to validate input URLs is complex enough that I think it would really help folks to do inline validation via custom js, rather than use the built-in browser validation. For instance, it took me quite some time to realize that the validation failure on my phone was due to the fact that my phone auto-capitalized the first letter of the URL. Without having access to the regex, it's tough to realize things are being rejected because you omitted the protocol, capitalized the protocol, have stray whitespace, or have selected an unsupported file-type or extensionless URL.

Probably not that much work to just use the same field-level validation pattern as is used for validating the URLs field in the new fact-checking UI... Presuming you are okay with the idea, I can send a PR to that effect in a bit.

Hi, thank you for your new and/or updated tools, this has been a very exciting day after your announcement! :)

I attempted to load a WARC from my machine on the website but it keeps getting stuck either at 15% or at 23%. The WARC file is relatively big (18MB), but not unreasonably so. It was made using the latest release of the Webrecorder Desktop app on Mac.

Feat. req. Load site`s start page on warc automatically.

To match Webrecorder Player functionality, double-clicking on these files should open ReplayWeb.page App in a new window

Hello! Replayweb.page app (v1.4.0—1.5.2 on Widndows 10 20H2) stops at 30-40% loading of any .warc file > 1GB from this save of gamerankings.com — https://archive.fart.website/archivebot/viewer/job/9uxhl
At the same time, 650MB file is loaded completely.
And your past program (Webrecorder player) loads any .warc file but is much slower.

Hi,
I've been experimenting with browsertrix-crawler and replayweb.page for a few weeks now, it's great!
However, I just encounter a problem with Replaying a warcz file generated by browsertrix-crawler.

Here is the problem: the top window is one of the warc file where I opened an image (original url of the image: https://img.atwikiimg.com/www65.atwiki.jp/sinoalice_kousatu/attach/40/212/%E3%82%A2%E3%83%AA%E3%82%B9%E8%A1%9D%E5%8B%9525.jpg)

And now the bottom window is the exact same image but from the warcz file (generated with the option --generateWACZ). This time, the image doesn't load.

Both the warc files and the warcz file are roughly the same size so I'm guessing the data is there.

Here's the exact command I used for generating the archive:
docker run -v $PWD/crawls:/crawls/ -it webrecorder/browsertrix-crawler crawl --url https://w.atwiki.jp/sinoalice_kousatu --include "https:\/\/w\.atwiki\.jp\/sinoalice_kousatu|https:\/\/img\.atwikiimg\.com" --ex clude "\/copy2\/|\/pedit\/|\/child\/|\/editx\/|\/renamex\/|\/editxx\/|\/chmod\/|\/chkind\/" --workers 3 --text --generateWACZ

Edit: I initially though it was because of the Japanese characters, but no, here's another example without any characters outside of the typical Latin character-set:

Edit2: here's the execution log in case there is something important there:

Text Extraction: Enabled
Load timeout for https://w.atwiki.jp/sinoalice_kousatu/pages/35.html TimeoutError: Navigation timeout of 90000 ms exceeded
    at /app/node_modules/puppeteer-core/lib/cjs/puppeteer/common/LifecycleWatcher.js:106:111

== Start:     2021-08-14 07:06:20.518
== Now:       2021-08-14 08:17:22.477 (running for 1.2 hours)
== Progress:  5198 / 5198 (100.00%), errors: 1 (0.02%)
== Remaining: 0.0 ms (@ 1.22 pages/second)
== Sys. load: 53.3% CPU / 90.8% memory
== Workers:   3
   #0 IDLE
   #1 IDLE
   #2 IDLE
Waiting to ensure pending data is written to WARCs...
Generating WACZ
Validating passed pages.jsonl file
Reading and Indexing All WARCs
Writing archives...
Generating page index from passed pages...
Header detected in the passed pages.jsonl file
Generating datapackage.json
Generating datapackage-digest.json

On mobile, input#filename escapes its container and the overflow is cut off. As well, since button[type="submit"].is-hidden-mobile, so far as I can tell, there is no way to submit the form from a mobile device, unless that device is using an external keyboard.

I spent a few minutes trying to fix while working on the aforementioned accessibility PR, but it was proving sufficiently finicky that I decided to make an issue instead. Might take another swing at it in a bit.

Hello. Is it possible to load and replay multiple warcs that are dependent on each other on this software? The warcs I've acquired are in this format and my only ability to replay them is limited to individual warcs, meaning co-dependent things like formatting and fonts and content from one warc are absent when loading the other. The closest I've come to finding a solution to this is to add all of them as a collection on webrecorder-desktop, but I would hope for a more practical and accessible method than using up space on the desktop's HDD and loading each warc individually.

Hello. When I am hosting the static page from a subdirectory and loading from, say, http://localhost:8989/replayweb.page/index.html, most behavior seems to work as expected. The only issue I can see is when clicking on a link within the rendered page and having it navigate to another archived page within that archive.

Here, the example 'netpreserve-twitter.warc' archive has a twitter page preserved that links to the blog post which is preserved. Clicking that link when serving the static pages from root works fine but it breaks when serving off a subpath after it tries to navigate to root when resolving the archive id and reloading. (I also wish it did not load the link in another tab and instead loaded within the iframe... but it doesn't seem to want to do that) I would expect it to resolve to /<subpath>/?source=blah, etc, and perhaps not reload.

I'm using Firefox 96 and my copy of Chrome just hangs and crashes when I load that twitter archive at both my local server and the main replayweb.page site using the warc file hosted there via 'source' query parameter. I don't know why. 😅

My goals would be to host replayweb.page as some kind of applet within my own web archive project that can embed artifacts in an iframe and it is rather important that it be off of a subpath and not trying to shoehorn my own project's root route. Thanks!

Hello team,

I was trying to run this repository and getting some warning and error and with these errors I am not able to move forward ,Please help me out if I am doing it with wrong way.

For the reference I am attaching a screenshot with this issue.

And when I type 'yes' to connect it anyway then I am getting these errors please find the below screenshot :-

Hi there,

first of all thank you very much for this great tool and the idea of the wacz format so simplify the handling of warc files.
I'm not sure if this is the correct project to report this too, so please let me know if I should go elsewhere.

I've greated a warc.gz file with grab-site. It's valid according to a check with warcio.

Using the warc-gz (4,3GB in size) works fine with replayweb.page if you're patient while it loads (using Chrome).

I created a wacz with

wacz create

It loads fine within replayweb.page.

Browsing through leads to an unexpected behaviour. Clicking on some links will bring you to a different page.
I crawled a bulletin board. So browsing a overview page and clicking on a thread will lead you to a differnt thread.

Example:
Overview page contains a link to showflat.php?Number=5970002, but it will take you to showflat.php?Number=1005878.

This is only happening with the wacz file, not with the warc.gz, so I'm pretty confident that it's either something in the wacz creation process, in the wabac.js handling or within replayweb.page.

I've verified that the the index.cdx file within the warc does contain an entry for showflat.php?Number=5970002.
The index.idx file does not, but I'm not sure if it should.

Looking at the IndexedDB via Chrome Dev Console shows 107813 entries for resources if I open the warc.gz file, just 2700 entries for the wacz file.

I'm happy to share the warc file if that helps or any other information that helps to debug this.

Downloaded the latest AppImage release 1.1.0 from GitHub, it doesn't start on Ubuntu 20.04.

Unfortunately there is not much meaningful debugging output:

despens@slice:~/bin$ chmod a+x ReplayWeb.page-1.1.0.AppImage 
despens@slice:~/bin$ ./ReplayWeb.page-1.1.0.AppImage 
bash: ./ReplayWeb.page-1.1.0.AppImage: No such file or directory

Double-clicking from the file manager doesn't launch a new task.

There are no issues with other AppImages on my system.

Hello! First off, thank you so much for creating this tool. It's really awesome and it's got me really excited about the future for web archiving.

While working on archiving a webpage for someone (here's the warc.gz I produced) I ran into an issue with Firefox (version 79.0b5 (64-bit)):

It got to the loading screen but never progressed past 0%. Since then the page has no longer responded to any clicks or user interaction, even after a reload.

It loads fine on Chromium (version 86.0.4185.0) however.

I was trying to debug the error but I couldn't figure out where it was coming from since there weren't any line numbers and I couldn't find the error message in this repository. I'm not sure the error message is even related to the freezup. Since I have a workaround I don't need to be unblocked, but I'd love out with making this project more robust if possible (if this is a user issue then sorry for wasting your time!).

For an in company use of webarchives, I'm experimenting transforming older Heritrix crawls to wacz (thanks to py-wacz). One of these transforms results in 48GB and reports 325.000 pages. Using this archive with the online replayweb.page results in out of memory. Opening it in the app (Windows 10) functions but not quite smooth.

I suppose the problem lies in the number of pages and/or the text index. What are the limitations for these?

Hi, I'm trying to read a very big WARC, of 18GB as I said in the title, and using the desktop version for Windows, the load stops (in fact, the app only change to a white screen, with the menus and the windows still active, so I supose it just make crash the algorithm of loading files) at the 21%, 4.2GBs in the progress bar, and 7.4GBs of effective RAM used for the app in that moment. I supose the app it's trying to load all the file in the RAM, so... It's something a bit troublesome when you are trying to use big files like this one (I have 16GBs of RAM btw, and well, when the load stops the total RAM was still at 80% aproximatelly, so isn't a problem of "no more RAM space", in this case at least). In older versions of this app, like the Web Recorder Player (x64), the WARC can be loaded without problems, except the app enter in an infinite "Indexing..." state and finally crash in a similar way when I try to access to the search field, but all the links are loaded in screen and I can access them if I just go to search them manually, so... I supose the problem isn't only the size of the file, but the way you manage to load the files. There is some way to fix this from my own? Or can you try to fix it in a next release?

Thank you.

Hello,

I have several WARCs that were made in another software. I would like to view them in Conifer, or replayweb.page. However, the warc’s cannot be displayed. They were made in a software called Capsis PreSurf, a company that no longer exists. These files were made in the time span of 2006-2015.

I have opened and checked the warcs, and they seem to conform to the WARC format standard. The files CAN indeed be displayed in the old viewer of Capsis PreSurf, which means that they are functional. However, they cannot be opened in replayweb.page or conifer.

Are there any known solutions to this, or can you tell me what the problem may be?

Thank you in advance.

André

Some crawlers could create multiple WARC files, it's importand if we had to upload WARC files to storages with limitation on single file size. I have a lot of archives websites splitted to 5-50 5GB WARC files each one.
Is it possible to add to Reply Web.Page ability to open more than one file at once ?

Hi,

Some days ago I created a WARC file with Heritrix. Webrecorder Players discovers around 10.000 pages; replayweb 0. There certainly are pages and URL's in that WARC-file. Is this a bug? Or maybe there is a dependency for the app that I had to install first?

I've created wacz file from warc.gz with latest py-warcz package 0.4.5
Original file https://cdn1.ruarxive.org/public/webcollect2022/ngo2022/cafrussia.ru/cafrussia.ru.warc.gz (179MB)
Produced WACZ file https://cdn1.ruarxive.org/public/webcollect2022/ngo2022/cafrussia.ru/cafrussia.ru.wacz (179MB)

I open wacz file with Reply Web.Page and I don't see it's contents.

There is no issues opening original warc.gz file.

Environment
OS - WIndows 10
Product version - reply web.page release 1.5.10 installed from binary from here https://github.com/webrecorder/replayweb.page/releases/tag/v1.5.10

I have a WARC file from a logged-in site that I generated using Webrecorder Desktop, and which replays correctly in Webrecorder and Webrecorder Player.

In replaywebpage, something (presumably JavaScript from the archive) is attempting to redirect to the logout page of the site, instead of displaying the original data from the archived session:

Sorry, the URL https://app.docusign.com/logout is not in this archive.

I've attached a sample .warc file (generated using a dummy account) for reproducing this bug.

Sample WARC file:

5nntovqatoa4bbxh-20201230001405.warc.zip

Click the "Bug Demo" bookmark.

What I expect to see:

Archived page is shown.

What I actually see:

"Sorry, the URL https://app.docusign.com/logout is not in this archive."

Software/Versions tested:

• OS: Debian GNU/Linux unstable (amd64)
• ReplayWeb.page-1.3.4.AppImage
• webrecorder-player-1.8.0.AppImage -- For "expected" screenshot.
• webrecorder-2.0.3.AppImage -- Used to create the WARC file.
• I also tried the web version today (2020-12-29) at https://replayweb.page, in both Chrome and Firefox. I did not notice any significant difference in behaviour compared to the AppImage version.

Issue can be possibly closed if Unpack option will be implemented

It would be nice to have a timeline similar to what web.archive.org has for pages there are multiple saved instances of.
It could be accessed by clicking on the displayed date field in the url bar, or some other way.

It would also be nice to be able to group saved pages in the search by url instead of one entry being shown for every version of a saved page.

Did I miss any built in functionality for this?

Whenever I try to access a page, the program fails and claims I tried to access a different page than the one I asked.

Example: I asked for http://127.0.0.1:8000/gdb/pt but I get an error message saying:

This page http://127.0.0.1:8000/gdb/ptnull is not part of this archive.

File I tried to load: biblia-disforia-genero-pt.har.txt Plase not that I had to change the file extension in order to upload it to GitHub. Just remove the final .txt and you should be fine.

Just documenting what I figured out to embed replayweb.page in a Ruby on Rails app (a Blacklight discovery implementation). Still working on our implementation of embedding WARCs, but in case anyone else runs into the same issues I did just getting it running I hope this is helpful.

I had a heck of a time getting Rails to register the service worker at ./replay/sw.js. First I tried registering the service worker as an object in the Rails app, but it turned out to be a red herring since the ui.js script really does all the heavy lifting. All I needed was to get the right path to sw.js which turned out to be tricky! I ended up using a gem called rails-static-router that allowed me to reroute the request for the sw.js file to the actual path on disk (in the Rails Public folder). So (using Rails 5):

/my-app/Gemfile

gem 'rails-static-router'

/my-app/config/routes.rb

get '/root-path/replay/sw.js', to: static('/replay/sw.js') # the `replay` folder is inside the app `Public` folder

That's it!

Hi, I'm having trouble loading archives over https in Firefox

Here is the archive i've been testing with: https://f.codl.fr/2201/test.warc.gz
and how I'm loading it into replayweb.page: https://replayweb.page/?source=https%3A%2F%2Ff.codl.fr%2F2201%2Ftest.warc.gz#url=https%3A%2F%2Ff.codl.fr%2Fdancing_baby.gif

When I load it into Chrome, it loads up fine. When I load it in Firefox, it fails with "An unexpected error occured: TypeError: NetworkError when attempting to fetch resource."

Firefox's console complains further about CORS:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://f.codl.fr/2201/test.warc.gz. (Reason: CORS request did not succeed). Status code: (null).

Failed to load ‘https://f.codl.fr/2201/test.warc.gz’. A ServiceWorker passed a promise to FetchEvent.respondWith() that rejected with ‘TypeError: NetworkError when attempting to fetch resource.’.

However I believe I am serving sufficient CORS headers per the documentation:

$ curl -Is "https://f.codl.fr/2201/test.warc.gz" | grep access-control    
access-control-allow-origin: *
access-control-allow-methods: GET, POST, OPTIONS, HEAD
access-control-expose-headers: Content-Encoding, Content-Range, Content-Length
access-control-allow-headers: *

When I wants to load a .warc file (6.03gb) loading stops at 6%.
With other files, this issue does not occur, but these files weigh less than 6gb
Need help, fast <3

I noticed that the netpreserve-twitter.warc example file has a list of pages and that it's possible to perform a full-text search on these pages. A WARC of a site created by wget doesn't have any pages defined and cannot be searched. I was hoping to somehow make this WARC searchable, but I have not succeeded in doing so. I tried converting the WARC to a WACZ using py-wacz, but the resulting WACZ file behaves exactly like the WARC file in repayweb.page.

I understand this is not really an issue with replayweb.page, but I was expecting to be able to use it's page search functionality on my WARC file just like with the example file. I've searched the net for information on creating a searchable WARC, but nothing has turned up.

We have a large .WARZ file sized over 25G which is not possible to be read by ReplayWeb with the following error messgae on screen:

Unexpected Loading Error: NotReadableError: The requested file could not be read, typically due to permission problems that have occurred after a reference to a file was acquired.

Hello and congratulations on all the new developments with the webrecorder tools.

I was trying out replayweb.page compared to the old Web Recorder Player, and the performance seems much improved. However, it looks like some images are not rendering for me in replayweb.page where they were working in the old player.

For example, here they are in the old player:

And here is the same page in the new one:

I am using the desktop application of both webrecorder player and replayweb.page, running in Windows 10.

Many thanks for all your work on this project.

I've been a big fan of webrecorder.io for a few years now and I've been using your tools Webrecorder Player, Webrecorder Desktop and the command line tool pywb extensively during that time.

The replaying peformance of Webrecorder Player has sadly always been quite bad for me, with complex JavaScript sites taking anywhere from 20-30 seconds to load despite the archive being stored on a local SSD (this was not the case with its predecessor WebArchivePlayer).

I gave this new tool a try just now and was blown away by the insane performance difference. The same page that takes over 20 seconds to replay in Webrecorder Player loads in less than two seconds with ReplayWeb.page:

In light of this, I wanted to ask if there are any plans to deprecate Webrecorder Player the same way it deprecated WebArchivePlayer and to focus development on ReplayWeb.page going forward.

P.S.: Thank you for all the work that you do! I immediately fell in love with the new embedding feature.

Hi @ikreymer @emmadickson

I know you guys are busy with WACZ but wanted to catch up with some issues we have been having on the embed version of replay web on Archipelago with version 1.2

I suspect a lot of this is because we are CDN loading the JS but also because the files we are testing agains are "largish" and also pure WARC. But we may have other issues so open to suggestions.

1. Safari (Version 14.0 (15610.1.28.1.9, 15610)) v/s Chrome (Version 86.0.4240.198 (Official Build) (x86_64)).
   This URL , a 1 GByte WARC file eventually loads on Safari (slow), I see small pauses made every 1000 records and I get a lot of restarts and failed attempts (inclusive from the Client reloading the whole page) and surprisingly loads faster (still a few minutes() and more resilient on Chrome but without any CSS/Images/assets

2. I updated the way we deliver the file to use streaming and was using a get argument (?stream=true) to enable modal during testing. This basically is read 1024 bytes from S3+ pass 1024 back to the HTTP request (I may chunk this larger?) Sadly can not use GET argument that in production because the embed tag fails if the "source" property is not just actual end in a valid file extension (not a big issue, modified it to always stream fro WARC and WACZ files in that first URL i shared). But know I have second thoughts. Is streaming what you need/works better? or is chunking better for your JS? Also, does the stream need to be seek-able?

To test a direct download of the stream please test this url (reusing or IIIF endpoint, please dismiss the weird semantic here)
https://webarchive.archipelago.nyc//do/4/iiif/51b281b4-093e-494c-9820-9eeeb03a4c6e/full/full/0/default.warc
e.g wget takes (942.50M 26.7MB/s in 39s ), replay embed 5 minutes of more on Chrome.

3. JS errors in every browser. We are getting quite a few. E.g Firefox:

failed to load ‘https://webarchive.archipelago.nyc/do/4/iiif/51b281b4-093e-494c-9820-9eeeb03a4c6e/full/full/0/default.warc’. A ServiceWorker intercepted the request and encountered an unexpected error. sw.js:33:126116
Read 1000 records sw.js:9:52746
AbortError: A request was aborted, for example through a call to IDBTransaction.abort. sw.js:9:179926
AbortError: AbortError

And It restarts. Sometimes it works, sometimes not.

E.g Safari

(anonymous function)
rejectPromise
rejectPromiseWithFirstResolvingFunctionCallCheck
s — sw.js:9:159182
s — sw.js:9:159182

Also should we be worried about the initial message (since running from CDN)

done
webarchive.archipelago.nyc/:8 GET https://webarchive.archipelago.nyc/replay/ui.js net::ERR_ABORTED 404 (Not Found)
ui.js:661 GET https://webarchive.archipelago.nyc/replay/wabac/api/id-a5343ec7bd53?all=1 404

We are loading ui.js via CDN and it works.

FYI we are running NGINX, and file delivery is not directly S3, (access control + some other users may be using Azure or directly filesystem so we wrap things. maybe we need to tune our Binary responses?

Sorry for the "cover it all" issue but I feel its more like a use case sharing and for sure using just WACZ should solve all the issues, but for now I want to be sure its not us/something we can do better

Thanks for the great work!!

When playing back web archives using ReplayWeb.page, we can get very high quality playback, and I think this is down to:

• how you index POST (and other non-GET???) requests.
• you fuzzy matching rules.

For large-scale web archives, we need to find ways to support this via OutbackCDX or SolrWayback/webarchive-discovery. Can you give us a clear summary of what we need to do ensure that pywb (and future ReplayWeb.page?) can achieve the same level of playback quality?

Hello, I am using the ReplayWebpage ArchiveWeb.page Chrome extension to capture webpages and analyze their network requests.

I have found some situations where parts of the POST data from the original request are not included in the requests of the replay.

steps to reproduce

1. Run this script (or clone this repo, run npm install, and run npm start.)

   const express = require('express')
   const app = express()
   const port = 3000
   
   app.get('/test.html', (req, res) => {
     res.send(`
       <html>
         <script>
           const request = new XMLHttpRequest()
           request.open('POST', '/whatever')
           request.setRequestHeader('Content-Type', 'application/json;charset=UTF-8')
           request.send(
             JSON.stringify([{ key: 'a' }, { key: 'b' }, { key: 'c' }])
           )
         </script>
         <body>
           Page loaded. Open the network tab and refresh the page.
         </body>
       </html>
     `)
   })
   
   app.listen(port, () => {
     console.log(`click here: http://localhost:${port}/test.html`)
   })
   

2. Navigate to http://localhost:3000/test.html in a web browser.
3. Verify that the request payload is:

   [{"key":"a"},{"key":"b"},{"key":"c"}]
   

4. Capture the webpage with the Chrome extension (or download the WARC from the repo.)
5. View the WARC at https://replayweb.page
6. Verify that the request payload is:

   ...localhost:3000/whatever?__wb_method=POST&key=c
   

So instead of having key0=a&key1=b&key2=c (or something similar) it only takes the last value, which in this case was "c."

The collection referenced in webrecorder/pywb#573 also causes replayweb.page to struggle. It takes over five minutes before the message Indexed 1234014 records is printed. Immediately thereafter, this warning is printed to the console:

DOMException: Transaction timed out due to inactivity.

My browser tab crashes soon thereafter. I'm using Chrome 80.0.3987.122, Linux.

This one takes a bit of explaining, so please bear with me 🙂.

As discussed at length with @ikreymer, we are working on a setup where users visit https://perma.cc/AAAA-AAAA and are served a page with a banner and an iframe. The iframe is pointed at https://replay.perma.cc, a page on a subdomain that includes an embedded playback of a warc using replayweb.page JS. It's working great in Chrome and Firefox, but does not work in Safari. (Example: https://perma.cc/7ESS-U63W?view-mode=client-side)

It is easy to reproduce outside of our setup: the following HTML just loads https://replayweb.page in an iframe.

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Replayweb.page in an iframe</title>
</head>
<body>
    <iframe style="width:100%; height: 100vh" src="https://replayweb.page">
</body>
</html>

If loaded in Firefox or Chrome, you can use the app normally, for instance, to request a playback of https://webrecorder-builds.s3.amazonaws.com/warcs/netpreserve-twitter.warc:

Observe the messages in the console: an initial 404 for https://replayweb.page/w/api/c/id-a3dec314b685?all=1, followed by the message that the service worker is now registered, followed by a message that 147 records were read, followed by nothing else. And, everything works, you can explore the archive and play back resources perfectly.

But, if loaded in Safari, this fetch request 404s again:

Observe the second 404 error in the console, after the message that 147 records were read. The response is indeed from the service worker, as opposed to the live web: {"error": "collection_not_found"}.

Something appears to be going wrong with IndexedDB.... and it has something to do with origins.

Both in the minimal reproduction above (HTML loaded into the browser directly with a file:/// protocol, iframe pointed at https://replayweb.page) and in my more complex setup (HTML served over SSL, iframe pointed at a subdomain), two instances of collDB appear in the Safari dev tools:

In all other browsers, CollDB only appears once. If I visit https://replayweb.page in Safari directly and request a playback of https://webrecorder-builds.s3.amazonaws.com/warcs/netpreserve-twitter.warc, collDB only appears once. AND.... if I change my setup so that my playback is not at a subdomain, but rather, both top-level page and the iframe are at the same origin.... then even in Safari CollDB only appears once, and even in Safari, the second fetch request succeeds

I am about out of ideas for how I might be able to fix, work around, or debug further.

• I have tried inspecting the file system to see if multiple sqlite DBs have been created: it doesn't look like it... but it's also not clear to me when Safari flushes this data to disk.

• I have added a log line to _init_DB and seen it print once and once only. I have created my own canary DBs in that function and observed that they, too, display twice in the Safari dev tools.

• I have created a canary DB elsewhere in the code, for instance, in this constructor, and seen that it does NOT display in duplicate in the Safari dev tools.

Any suggestions?

Note: if you experiment with the minimal reproduction above, you'll see an error to the tune of Blocked a frame with origin "file:///...." from accessing a frame with origin "https://replayweb.page". Protocols, domains, and ports must match.: while suggestive, that's not actually the problem. The complete repro, where the iframe's src url has the CORS headers Access-Control-Allow-Origin=* and Access-Control-Allow-Methods=*, does not therefore throw that error, and yet behaves exactly as described.

When trying to load a CDX (local or remote), despite this doc saying it's supported, it shows a "Please match the requested format" and refuses to load the CDX.

Using warc2zim, I created a ZIM from a WARC (created using conifere) that you can find at http://tmp.kiwix.org/warc2zim/

There is a styling problem on the iframe that makes the bottom of the pages unreachable and invisible.

Actual content is visible at https://isago.ml

It's not about the code, but could be related to the version of IPFS on the site. If the ipfs://xxxx is not offered as Qm****, instead as ba**** (CID format), it won't show the content from IPFS network.

e.g.
ipfs://bafybeia5s7bazmnyo3zvqgiuwu3airnlugicyp44dj5w6pfeiakja3iqfu/hkcnews.wacz is not shown in the replayweb.page.

Hi,

I have a WARC-file of 970 MB. I discovered that the full archive is not loaded, not even 1 MB of the archive is loaded. A lot of pages/URL's can't be seen with replayweb.page.

My pages also have no styling.

Maybe I'm doing something wrong? the WARC was created with Heritrix.

I loaded a pretty large file (40Gb roughly) and when I tried to search the whole archive by keywords I used url prefix search. But the results contain links that have been loaded only. Is there a way to find urls that haven't been loaded yet?

Following up on openzim/zimit#71.

Zimit 1.1.4 uses browsertrix-crawler:0.3.1 and [email protected].

With those, we seem to capture Youtube embeds as there are matching files in the WARC/ZIM (hard to tell actually) that piles up to 800MB+

Here's a few of those entries in the ZIM, sorted by decreasing sizes:

First thing you'd notice is that there are multiple entries with the very same bytesize, for all of the entries. Safe to assume it's not a coincidence and we are bundling each video data 3+ times each.
Note that the log is full of revisit entries.

Now that's not the issue I'm after here. The problem is that those video don't replay. I'm not sure what kind of feedback I should provide here. Please let me know. You can download the ZIM from here (expires in a week).

Source URL: https://mesquartierschinois.wordpress.com/

Hey, I don't know if it's possible, but I have a rather large WARC (1.3 Gigabytes for a crawl of a probably 1mb site), and I want to see where the culprits of the space are. Can I sort a WARC by its size?

from doc:

Thus, if the HTML snippet was added to https://my-site.example.com/path/my-web-archive-embed.html then the sw.js should be added such that it is at: https://my-site.example.com/path/replay/sw.js.

i have this example: https://docuver.se/replayweb/mlol-iiif-stories.html

the sw in this path is not working https://docuver.se/replayweb/replay/sw.js

ui.js:199 Error during service worker registration: TypeError: Failed to register a ServiceWorker for scope ('https://docuver.se/replay/') with script ('https://docuver.se/replay/sw.js'): A bad HTTP response code (404) was received when fetching the script.

while it works if saved on https://docuver.se/replay/sw.js