1

I have a small problem. I want to save web pages using Firefoxs "SAVE" button and then click the "Next" button to the next page and save it. In total I need to store 1000 pages.

Apparently I can't do it manually. It takes ages. The web page can't be downloaded by wget because they aren't simple html files. They use JavaScript files. Any good ideas?

Raphael Ahrens
  • 9,771
Mrk421
  • 11
  • 2
  • @Tejas that is not a duplicate, the OP does explicitly not ask for submitting the page, but for downloading a page that is (partly) build by javascript. – Anthon Jan 02 '15 at 08:17

1 Answers1

1

What "always" works is using selenium from some programming language. You do of course face the same problem that humans face when browsing of javascript driven files: you never know if it is fully finished building up the page.

You will not be able to use the Ctrl+S or the SaveButton, but selenium does allow you to retrieve the html of the page. Normally it is good enough to wait X seconds (X depending on the complexity of building up the page, and your connection). You can also check for elements that should be available in "completed" page (i.e. after enough JavaScript building of the DOM tree has finished) and save the HTML based on that trigger.

Anthon
  • 79,293