Concurrency

Create a class named AsyncCrawler to manage shared state for your crawler. Create an __init__ method to initialize the following fields:

• base_url (the starting URL)
• base_domain (the domain name)
• page_data (our dictionary of page data, keyed by normalized URL)
• visited (a set of normalized URLs already scheduled for crawling)
• lock (an asyncio.Lock to safely update page_data)
• max_concurrency (to limit the number of requests allowed at once)
• semaphore (an asyncio.Semaphore - pass it the value of max_concurrency)
• session (an aiohttp.ClientSession for making HTTP requests)

Write two new async methods for our AsyncCrawler named __aenter__ and __aexit__ to open and close the aiohttp session:

async def __aenter__(self):
    self.session = aiohttp.ClientSession()
    return self


async def __aexit__(self, exc_type, exc_val, exc_tb):
    await self.session.close()

Add an async add_page_visit method to your AsyncCrawler.

async def add_page_visit(self, normalized_url):

• Take a normalized URL
• Use async with on the lock field to safely check and update visited
• If the normalized URL is already in visited, return False
• Otherwise, add it to visited and return True before fetching the page

Make your old get_html function an async method of AsyncCrawler.

• Use the aiohttp client you set in the session field along with async with to fetch the page.
• Keep the same error, status, and header checks

Update your old crawl_page function and make it an async method of AsyncCrawler.

• Call and await your new add_page_visit method. If it isn't a new page, return early
• Use async with self.semaphore to limit the number of concurrent requests
• Fetch the page's HTML and extract page data using extract_page_data
• Add the page data to page_data dictionary using the normalized URL as the key (use the lock to do this safely)
• Use the extracted page data's outgoing_links as the URLs to crawl next
• For each URL, create a task to crawl it using asyncio.create_task
• Wait for all tasks with await asyncio.gather(*tasks)

Add an async crawl method to AsyncCrawler that starts by crawling base_url and returns the page_data dictionary

Write an async function named crawl_site_async. Make sure it:

• Creates and uses an AsyncCrawler in an async with block
• Calls/awaits crawl() and returns the final page_data dictionary

Update main.py to use crawl_site_async

• Instead of calling get_html, call crawl_site_async(base_url) and await the results
• When iterating over the page data, use page_data.values() to get the page dictionaries
• Update the if __name__ == "__main__" block to run your async main function using asyncio.run(main())

Test it (I tested it manually again). I recommend setting your max_concurrency (the value given to your semaphore) to 1 to start to just make sure everything still works. Then bump it up to 3, 5, or 10 to see it run faster.