Skip to content

Added intercept request example #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Added intercept request example #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 21 additions & 20 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,26 +35,27 @@ The following examples are currently available:

<!-- the following section is updated by running `go run gen.go` -->
<!-- START EXAMPLES -->
| Example | Description |
|-----------------------------------|------------------------------------------------------------------------------|
| [click](/click) | use a selector to click on an element |
| [cookie](/cookie) | set a HTTP cookie on requests |
| [download_file](/download_file) | do headless file downloads |
| [download_image](/download_image) | do headless image downloads |
| [emulate](/emulate) | emulate a specific device such as an iPhone |
| [eval](/eval) | evaluate javascript and retrieve the result |
| [headers](/headers) | add extra HTTP headers to browser requests |
| [keys](/keys) | send key events to an element |
| [logic](/logic) | more complex logic beyond simple actions |
| [pdf](/pdf) | capture a pdf of a page |
| [proxy](/proxy) | authenticate a proxy server which requires authentication |
| [remote](/remote) | connect to an existing Chrome DevTools instance using a remote WebSocket URL |
| [screenshot](/screenshot) | take a screenshot of a specific element and of the entire browser viewport |
| [submit](/submit) | fill out and submit a form |
| [subtree](/subtree) | populate and travel a subtree of the DOM |
| [text](/text) | extract text from a specific element |
| [upload](/upload) | upload a file on a form |
| [visible](/visible) | wait until an element is visible |
| Example | Description |
|-----------------------------------------|------------------------------------------------------------------------------|
| [click](/click) | use a selector to click on an element |
| [cookie](/cookie) | set a HTTP cookie on requests |
| [download_file](/download_file) | do headless file downloads |
| [download_image](/download_image) | do headless image downloads |
| [emulate](/emulate) | emulate a specific device such as an iPhone |
| [eval](/eval) | evaluate javascript and retrieve the result |
| [headers](/headers) | add extra HTTP headers to browser requests |
| [intercept_request](/intercept_request) | do headless intercept request to download files |
| [keys](/keys) | send key events to an element |
| [logic](/logic) | more complex logic beyond simple actions |
| [pdf](/pdf) | capture a pdf of a page |
| [proxy](/proxy) | authenticate a proxy server which requires authentication |
| [remote](/remote) | connect to an existing Chrome DevTools instance using a remote WebSocket URL |
| [screenshot](/screenshot) | take a screenshot of a specific element and of the entire browser viewport |
| [submit](/submit) | fill out and submit a form |
| [subtree](/subtree) | populate and travel a subtree of the DOM |
| [text](/text) | extract text from a specific element |
| [upload](/upload) | upload a file on a form |
| [visible](/visible) | wait until an element is visible |
<!-- END EXAMPLES -->

## Contributing
Expand Down
122 changes: 122 additions & 0 deletions intercept_request/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
// Command intercept_request is a chromedp example demonstrating how to do
// headless intercept request to download files. Useful if the download is trigger in a new tab
//
package main

import (
"context"
"fmt"
"github.com/chromedp/cdproto/browser"
"github.com/chromedp/cdproto/cdp"
"github.com/chromedp/cdproto/fetch"
"io/ioutil"
"log"
"net/http"
"strings"
"time"

"github.com/chromedp/cdproto/network"
"github.com/chromedp/chromedp"
)

func main() {
// create context
ctx, cancel := chromedp.NewContext(
context.Background(),
chromedp.WithLogf(log.Printf),
)
defer cancel()

// create a timeout as a safety net to prevent any infinite wait loops
ctx, cancel = context.WithTimeout(ctx, 60*time.Second)
defer cancel()

// set up a channel, so we can block later while we monitor the download
// progress
done := make(chan bool)

// set up a listener to watch the fetch events and close the channel when
// expected event are fired and the file is downloaded
chromedp.ListenTarget(ctx, func(v interface{}) {
switch ev := v.(type) {
case *fetch.EventRequestPaused:
go func() {
c := chromedp.FromContext(ctx)
e := cdp.WithExecutor(ctx, c.Target)

// find the matching request to intercept it.
// The request will be blocked and processed with go http client
if strings.HasSuffix(ev.Request.URL, "tar.gz") {
if err := fetch.FailRequest(ev.RequestID, network.ErrorReasonBlockedByClient).Do(e); err != nil {
log.Printf("Failed to abort request: %v", err)
}

// pass the request
downloadFile(ev.Request)

// we're done, closing the channel
done <- true
} else {
// continue others events
if err := fetch.ContinueRequest(ev.RequestID).Do(e); err != nil {
log.Printf("Failed to continue request: %v", err)
}
}
}()
}
})

// navigating to the latest chromedp releases and download
if err := chromedp.Run(ctx,
fetch.Enable(),
// not needed because the download request is blocked but kept to illustration that is really no download
browser.SetDownloadBehavior(browser.SetDownloadBehaviorBehaviorDeny),
chromedp.Navigate("https://github.com/chromedp/chromedp/releases/latest"),
chromedp.Click("//a[contains(@href,'tar.gz')]"),
); err != nil {
log.Fatal(err)
}

// This will block until the chromedp listener closes the channel
<-done
}

func downloadFile(r *network.Request) {
var req *http.Request
var err error
if r.HasPostData {
req, err = http.NewRequest(r.Method, r.URL, strings.NewReader(r.PostData))
} else {
req, err = http.NewRequest(r.Method, r.URL, nil)
}

if err != nil {
log.Fatalf("failed to create http client: %v\n", err)
}

// adding all request headers including cookies
for key, val := range r.Headers {
req.Header.Add(key, fmt.Sprintf("%v", val))
}

client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Fatalf("failed to send http request: %v\n", err)
}

// expect status code 200
if resp.StatusCode != http.StatusOK {
log.Fatalf("bad status: %s\n", resp.Status)
}

defer resp.Body.Close()
buf, err := ioutil.ReadAll(resp.Body)

// write the file to disk - since we hold the bytes we dictate the name and
// location
if err := ioutil.WriteFile("source.tar.gz", buf, 0644); err != nil {
log.Fatal(err)
}
log.Print("wrote source.tar.gz")
}