read_html() operates on the HTML source code downloaded from the server.
This works for most websites but can fail if the site uses javascript to
generate the HTML. read_html_live() provides an alternative interface
that runs a live web browser (Chrome) in the background. This allows you to
access elements of the HTML page that are generated dynamically by javascript
and to interact with the live page by clicking on buttons or typing in
forms.
read_html_live() requires Chrome to be installed and is considerably
slower and more resource intensive than read_html(), so we recommend
using read_html() by default and only switching to read_html_live() if
the data you want isn't present in the static HTML, or you need to
interact with the page (e.g. clicking buttons or filling in forms).
Behind the scenes, this function uses the chromote package, which requires that you have a copy of Google Chrome installed on your machine.
Arguments
- url
Website url to read from.
- mode
Either
"headless"(the default) to run Chrome without a visible window, or"visible"to open a browser window, which can be useful when debugging a scraping script.- view
Either
"desktop"(the default) or"mobile", controlling the viewport size that the page is rendered with.- browser
An existing chromote::Chromote browser object to use. By default, all sessions share a single browser (per
mode) that is launched on first use and closed when the package is unloaded; each session gets its own tab. Supply your own browser if you need full control over its lifecycle.- timeout
Number of seconds to wait for the page to finish loading. You may need to increase this if you're using a slow proxy.
Value
read_html_live() returns an R6 LiveHTML object. You can interact
with this object using the usual rvest functions, or call its methods,
like $click(), $scroll_to(), and $type() to interact with the live
page like a human would.
Examples
if (FALSE) { # \dontrun{
# When we retrieve the raw HTML for this site, it doesn't contain the
# data we're interested in:
static <- read_html("https://www.forbes.com/top-colleges/")
static |> html_element("table")
# Instead, we need to run the site in a real web browser, causing it to
# download a JSON file and then dynamically generate the html:
dynamic <- read_html_live("https://www.forbes.com/top-colleges/")
# You may need to click the cookie consent banner if it appears
dynamic$view()
# Now we can find the table
dynamic |> html_element("table")
# And extract data from it
dynamic |>
html_element("table") |>
html_table()
} # }