Example scraping for Twitter needed

[BrandonTay2001]

Hi all, have been playing around with Crawl4Ai and has been great so far. Wanted to know if anyone has experience using this for Twitter profiles or posts. Here's what I tried:

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, JsonCssExtractionStrategy

async def main():

    browser_config = BrowserConfig(
        headless=True,
        cookies=[
            {"name": "auth_token", "value": "some_value", "domain": ".x.com", "path": "/"}
        ],
        verbose=True,
        extra_args=['--disable-web-security']
    )

    crawler_config = CrawlerRunConfig(
        delay_before_return_html=4.0,
        magic=True,
        only_text=True,
        simulate_user=True,
    )

    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://x.com/user/status/id",
            run_config=crawler_config,
        )
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())

From experience, I'm able to get some gibberish which just seems to be the header in Twitter. This is what I obtained:

## 
To view keyboard shortcuts, press question mark
[View keyboard shortcuts](https://x.com/i/keyboard_shortcuts)
# [](https://x.com/home)
[](https://x.com/home)[](https://x.com/explore)[](https://x.com/notifications)[](https://x.com/messages)[](https://x.com/i/grok)[](https://x.com/user)
[](https://x.com/compose/post)
![username](https://pbs.twimg.com/profile_images/profilePic.jpg)

Not sure if this is a bug or something I'm doing wrong, so refraining from opening an issue for now. Much appreciated!