FireCrawl

Maker

Type

About

These nodes provide web crawling and web scraping functionality to systematically browse and collect data from websites with the help of Firecrawler.

FireCrawl is an application that provides web crawling and web scraping capabilities, allowing users to systematically browse and collect data from websites. With FireCrawl, you can create crawl jobs to be notified of all crawl steps and instantly scrape any URL. The application is highly configurable, offering options to include or exclude specific paths and HTML tags, emulate mobile devices, and manage page load times. Integrated with Lamatic.ai, FireCrawl offers users a flexible tool for gathering and processing web data efficiently.

What can you do?

Build a system to monitor changes on specific web pages by creating crawl jobs with webhook notifications for each step.
Develop a data extraction tool that scrapes web pages with highly configurable parameters, allowing for the inclusion or exclusion of specific HTML tags.
Implement a mobile-compatible web scraping solution by emulating mobile devices during the scraping process.
Create an automated content aggregator that collects and processes web data efficiently, tailored to specific requirements and configurations.

Available Functionality

Actions

✅ Create Crawl Webpage Job

✅ Scrape Webpage Instantly

Install Steps

Drag node to the workflow.
Click on Add new Credentials.
Follow the Install Steps.
Once added choose the new credentials.

Configuration

Select either Crawler Node or Scraper Node.
Name - Display Name for the Node.
Credentials - Add API key and hostUrl (we do support self hosted firecrawl instances) for your Firecrawl instance.
For Crawler

Add the URL you want to crawl, you can also select it from variable selector.
Set the Notification Webhook where you want to be notified of all actions and get information when crawl job ends.
If you want to Crawl sub-pages, toggle the switch (optional).
Set the Limit of number of pages to Crawl.
Set the depth that is how many nested path the crawler should crawl till.
Additionally if you want to exclude or include specific parts add them.

For Scraper

Add the URL you want to scrape, you can also select it from variable selector.
If you want to scrape only content, in main tag set the toggle switch.
Similarly toggle switch if you want to emulate mobile device and skipTLS verification respectively.
If your target URL takes time to load, set wait for page load accordingly.
Additionally if you want to exclude or include specific HTML tags, add them.

LowCode Config

// Low Code Config for Crawler Node
  - nodeId: crawlerNode_360
    nodeType: crawlerNode
    nodeName: Crawler
    values:
      credentials: FireCrawl Credentials
      url: 'url to be crawled'
      webhook: >-
        'your webhook url'
      crawlSubPages: true
      crawlLimit: 10
      crawlDepth: 1
      excludePath: []
      includePath: []
      
      
  //Low Code Config for Scraper Node 
  
  - nodeId: scraperNode_982
    nodeType: scraperNode
    nodeName: Scraper
    values:
      credentials: FireCrawl Credentials
      url: 'url to be crawled'
      onlyMainContent: true
      skipTLsVerification: false
      mobile: false
      waitFor: 123
      includeTags: []
      excludeTags: []
    needs:
      - triggerNode_1

Example Workflow Setup

Scraper:

Crawler:

Output

// Crawler Output

 "output": {
    "success": true,
    "id": "crawlID",
    "url": "baseUrl/v1/crawl/crawlId"
  },


// Scraper Output

 "output": {
    "markdown": "![Lamatic](https://studio.lamatic.ai/logo.svg)\n\n### Create your account\n\nContinue with GitHub![Google](https://studio.lamatic.ai/google.svg)Continue with Google\n\nor\n\nEmail\n\nPassword\n\nSign Up\n\nHave an account? [Log In](/login)\n\nBy continuing, you agree to Lamatic's [Terms of Service](https://lamatic.ai/docs/legal/terms-of-service) and [Privacy Policy](https://lamatic.ai/docs/legal/privacy-policy), and to receive periodic emails with updates.\n\n> My first reaction was 'this is too good to be true, it has to be a gimmick or vaporware' ... you're definitely onto something!\n\n![Joe](https://studio.lamatic.ai/quotes/joe.png)\n\nJoe\n\nCEO • Daily.ai",
    "html": "<!DOCTYPE html><html lang=\"en\" class=\"__variable_2b24f9 __variable_57c479 light\" style=\"color-scheme: light;\"><body><next-route-announcer style=\"position: absolute;\"></next-route-announcer><div class=\"flex relative min-h-screen bg-white\"><div class=\"absolute top-6 left-4\"><img src=\"/logo.svg\" alt=\"Lamatic\"></div><div class=\"flex-1 h-screen flex justify-center items-center\"><div class=\"rounded-lg p-8 bg-card text-card-foreground flex w-[600px] justify-center flex-col m-4 border-none shadow-none\"><div class=\"flex flex-col space-y-1.5 p-6\"><h3 class=\"tracking-tight text-2xl font-semibold\">Create your account</h3></div><div class=\"p-6 pt-0 space-y-4 mt-3\"><div class=\"space-y-4\"><button class=\"inline-flex items-center justify-center whitespace-nowrap rounded-md font-medium ring-offset-background transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:pointer-events-none disabled:opacity-50 border border-input bg-white hover:bg-secondary hover:text-accent-foreground px-4 py-2 w-full h-14 text-lg\"><svg width=\"15\" height=\"15\" viewBox=\"0 0 15 15\" fill=\"none\" xmlns=\"http://www.w3.org/2000/svg\" class=\"mr-2 h-6 w-6\"><path d=\"M7.49933 0.25C3.49635 0.25 0.25 3.49593 0.25 7.50024C0.25 10.703 2.32715 13.4206 5.2081 14.3797C5.57084 14.446 5.70302 14.2222 5.70302 14.0299C5.70302 13.8576 5.69679 13.4019 5.69323 12.797C3.67661 13.235 3.25112 11.825 3.25112 11.825C2.92132 10.9874 2.44599 10.7644 2.44599 10.7644C1.78773 10.3149 2.49584 10.3238 2.49584 10.3238C3.22353 10.375 3.60629 11.0711 3.60629 11.0711C4.25298 12.1788 5.30335 11.8588 5.71638 11.6732C5.78225 11.205 5.96962 10.8854 6.17658 10.7043C4.56675 10.5209 2.87415 9.89918 2.87415 7.12104C2.87415 6.32925 3.15677 5.68257 3.62053 5.17563C3.54576 4.99226 3.29697 4.25521 3.69174 3.25691C3.69174 3.25691 4.30015 3.06196 5.68522 3.99973C6.26337 3.83906 6.8838 3.75895 7.50022 3.75583C8.1162 3.75895 8.73619 3.83906 9.31523 3.99973C10.6994 3.06196 11.3069 3.25691 11.3069 3.25691C11.7026 4.25521 11.4538 4.99226 11.3795 5.17563C11.8441 5.68257 12.1245 6.32925 12.1245 7.12104C12.1245 9.9063 10.4292 10.5192 8.81452 10.6985C9.07444 10.9224 9.30633 11.3648 9.30633 12.0413C9.30633 13.0102 9.29742 13.7922 9.29742 14.0299C9.29742 14.2239 9.42828 14.4496 9.79591 14.3788C12.6746 13.4179 14.75 10.7025 14.75 7.50024C14.75 3.49593 11.5036 0.25 7.49933 0.25Z\" fill=\"currentColor\" fill-rule=\"evenodd\" clip-rule=\"evenodd\"></path></svg>Continue with GitHub</button><button class=\"inline-flex items-center justify-center whitespace-nowrap rounded-md font-medium ring-offset-background transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:pointer-events-none disabled:opacity-50 border border-input bg-white hover:bg-secondary hover:text-accent-foreground px-4 py-2 w-full h-14 text-lg\"><img src=\"/google.svg\" alt=\"Google\" class=\"mr-2 h-6 w-6\">Continue with Google</button></div><div class=\"relative py-4\"><div class=\"absolute inset-0 flex items-center\"><div class=\"w-full border-t border-gray-300\"></div></div><div class=\"relative flex justify-center\"><span class=\"px-3 bg-white font-medium\">or</span></div></div><form class=\"space-y-6\"><div class=\"space-y-1\"><label class=\"text-sm font-medium leading-none peer-disabled:cursor-not-allowed peer-disabled:opacity-70\" for=\"email\">Email</label><input class=\"flex h-10 w-full rounded-md border border-input bg-background px-3 py-2 text-sm ring-offset-background file:border-0 file:bg-transparent file:text-sm file:font-medium placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:cursor-not-allowed disabled:opacity-50\" id=\"email\" placeholder=\"Enter your email\" required=\"\" type=\"email\" value=\"\"></div><div class=\"space-y-1 relative\"><label class=\"text-sm font-medium leading-none peer-disabled:cursor-not-allowed peer-disabled:opacity-70\" for=\"password\">Password</label><div class=\"relative\"><input class=\"flex h-10 w-full rounded-md border border-input bg-background px-3 py-2 text-sm ring-offset-background file:border-0 file:bg-transparent file:text-sm file:font-medium placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:cursor-not-allowed disabled:opacity-50\" id=\"password\" placeholder=\"Enter your password\" required=\"\" type=\"password\" value=\"\"></div></div><button class=\"inline-flex items-center justify-center whitespace-nowrap rounded-md text-sm font-medium ring-offset-background transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:pointer-events-none disabled:opacity-50 bg-primary border border-secondary/90 text-primary-foreground hover:opacity-85 h-10 px-4 py-2 w-full\" type=\"submit\">Sign Up</button></form></div><div class=\"flex p-6 pt-0 flex-col items-start space-y-2\"><p class=\"w-full text-center text-muted-foreground\">Have an account?<a class=\"ml-1 text-foreground font-medium cursor-pointer hover:underline\" href=\"/login\">Log In</a></p><p class=\"!mt-10 px-8 text-sm text-center text-muted-foreground\">By continuing, you agree to Lamatic's <a class=\"underline\" target=\"_blank\" href=\"https://lamatic.ai/docs/legal/terms-of-service\">Terms of Service</a> and <a class=\"underline\" target=\"_blank\" href=\"https://lamatic.ai/docs/legal/privacy-policy\">Privacy Policy</a>, and to receive periodic emails with updates.</p></div></div></div><div class=\"hidden md:flex md:flex-1 bg-muted items-center justify-center p-8\"><div class=\"max-w-md\"><svg width=\"50\" height=\"42\" viewBox=\"0 0 50 42\" fill=\"none\" xmlns=\"http://www.w3.org/2000/svg\"><path d=\"M48.375 41.625H28.25V21.875L36.125 0.5H49.125L36.5 21.875H48.375V41.625ZM20.25 41.625H0V21.875L8 0.5H20.875L8.25 21.875H20.25V41.625Z\" fill=\"#EAE3D7\"></path></svg><blockquote class=\"text-2xl font-bold my-12\">My first reaction was 'this is too good to be true, it has to be a gimmick or vaporware' ... you're definitely onto something!</blockquote><div class=\"flex items-center\"><span class=\"relative flex shrink-0 overflow-hidden rounded-full h-10 w-10 mr-4\"><img class=\"aspect-square h-full w-full\" alt=\"Joe\" src=\"/quotes/joe.png\"></span><div><p class=\"font-medium\">Joe</p><p class=\"text-sm text-muted-foreground\">CEO • Daily.ai</p></div></div></div></div></div><div role=\"region\" aria-label=\"Notifications (F8)\" tabindex=\"-1\" style=\"pointer-events: none;\"><ol tabindex=\"-1\" class=\"fixed top-0 left-1/2 z-[100] flex max-h-screen w-full -translate-x-1/2 flex-col p-4 sm:top-0 sm:right-auto sm:left-1/2 sm:flex-col md:max-w-[580px]\"></ol></div><div id=\"crisp-loader\"></div></body></html>",
    "title": "Lamatic.ai | Studio",
    "description": "DevOps Platform for GenAI",
    "language": "en",
    "ogLocaleAlternate": [],
    "viewport": "width=device-width, initial-scale=1",
    "sourceURL": "https://studio.lamatic.ai",
    "statusCode": 200
  },

Troubleshooting Common Issues

‣

Indexing Issues:

‣

Webhook Data Issues