│ │ Comprehensive Cloudflare Settings Audit for Search Engine Indexing Optimization │ │
│ │ │ │
│ │ Executive Summary │ │
│ │ │ │
│ │ After successfully fixing the Kuyadoga indexing issue by disabling Browser Integrity Check, I need to audit all Cloudflare settings that could potentially block search engine crawlers from properly │ │
│ │ indexing your websites. This comprehensive audit will cover both kuyadoga.com and malla-supply.com to ensure optimal search engine accessibility. │ │
│ │ │ │
│ │ Phase 1: Critical Security Settings Review │ │
│ │ │ │
│ │ 1.1 Bot Fight Mode & Super Bot Fight Mode │ │
│ │ │ │
│ │ - Location: Security → Bots │ │
│ │ - Current Issue: Can block legitimate search engine crawlers │ │
│ │ - Action: Verify Bot Fight Mode is OFF for kuyadoga.com, review for malla-supply.com │ │
│ │ - For Premium Plans: Configure "Super Bot Fight Mode" to ALLOW "Definitely automated" bots (Googlebot, Bingbot) │ │
│ │ │ │
│ │ 1.2 Browser Integrity Check │ │
│ │ │ │
│ │ - Location: Security → Settings │ │
│ │ - Status: ✅ Already fixed for kuyadoga.com (OFF) │ │
│ │ - Action: Verify malla-supply.com setting is optimal (keep ON since it's working) │ │
│ │ │ │
│ │ 1.3 Security Level Settings │ │
│ │ │ │
│ │ - Location: Security → Settings │ │
│ │ - Current Risk: High/Under Attack modes can block crawlers │ │
│ │ - Action: Ensure set to "Medium" or lower, never use "I'm Under Attack" permanently │ │
│ │ - Impact: Challenge pages prevent bots from crawling content │ │
│ │ │ │
│ │ 1.4 Challenge Passage │ │
│ │ │ │
│ │ - Location: Security → Settings │ │
│ │ - Risk: Can delay or block search engine crawlers │ │
│ │ - Action: Verify not set to aggressive challenge modes │ │
│ │ │ │
│ │ Phase 2: Web Application Firewall (WAF) Configuration │ │
│ │ │ │
│ │ 2.1 WAF Custom Rules │ │
│ │ │ │
│ │ - Location: Security → WAF → Custom Rules │ │
│ │ - Critical Action: Create "Allow Verified Bots" rule │ │
│ │ - Rule Expression: (cf.client.bot) or (cf.verified_bot_category in {"Search Engine Crawler"}) │ │
│ │ - Action: Skip all security checks for verified search engine bots │ │
│ │ │ │
│ │ 2.2 WAF Managed Rules │ │
│ │ │ │
│ │ - Location: Security → WAF → Managed Rules │ │
│ │ - Risk: May contain rules that block legitimate crawlers │ │
│ │ - Action: Review for any rules blocking search engine user agents or IPs │ │
│ │ │ │
│ │ 2.3 Rate Limiting Rules │ │
│ │ │ │
│ │ - Location: Security → WAF → Rate Limiting │ │
│ │ - Critical: Exclude verified bots from rate limiting │ │
│ │ - Action: Add exception for cf.verified_bot_category in {"Search Engine Crawler"} │ │
│ │ │ │
│ │ Phase 3: Access Control Audit │ │
│ │ │ │
│ │ 3.1 IP Access Rules │ │
│ │ │ │
│ │ - Location: Security → WAF → Tools → IP Access Rules │ │
│ │ - Risk: May have accidentally blocked Google IP ranges │ │
│ │ - Action: Check for any blocked IPs that could include search engine crawlers │ │
│ │ - Google IP Ranges: Verify none are in block list │ │
│ │ │ │
│ │ 3.2 User Agent Blocking │ │
│ │ │ │
│ │ - Location: Security → WAF → Tools → User Agent Blocking │ │
│ │ - Risk: May block legitimate crawler user agents │ │
│ │ - Action: Remove any rules blocking "Googlebot", "Bingbot", etc. │ │
│ │ - Recommendation: Replace with custom rules using bot verification │ │
│ │ │ │
│ │ 3.3 Country/Geographic Blocking │ │
│ │ │ │
│ │ - Location: Security → WAF → Tools → IP Access Rules │ │
│ │ - Risk: May block crawlers from certain countries │ │
│ │ - Action: Ensure no blanket country blocks that affect major search engines │ │
│ │ │ │
│ │ Phase 4: Page Rules & Configuration Rules │ │
│ │ │ │
│ │ 4.1 Page Rules Review │ │
│ │ │ │
│ │ - Location: Rules → Page Rules │ │
│ │ - Check For: Any rules that might affect crawler access │ │
│ │ - Security Level: Ensure no pages set to "High" or "Under Attack" │ │
│ │ - Browser Integrity Check: Verify no conflicting page-level settings │ │
│ │ │ │
│ │ 4.2 Configuration Rules │ │
│ │ │ │
│ │ - Location: Rules → Configuration Rules │ │
│ │ - Action: Review any rules affecting security settings │ │
│ │ - Ensure: No rules that might inadvertently block crawlers │ │
│ │ │ │
│ │ Phase 5: Advanced Settings & Monitoring │ │
│ │ │ │
│ │ 5.1 DDoS Protection │ │
│ │ │ │
│ │ - Location: Security → DDoS │ │
│ │ - Action: Ensure legitimate crawlers aren't caught in DDoS mitigation │ │
│ │ - Recommendation: Use "Managed Challenge" instead of "Block" where possible │ │
│ │ │ │
│ │ 5.2 Security Events Monitoring │ │
│ │ │ │
│ │ - Location: Security → Events │ │
│ │ - Action: Set up monitoring for blocked search engine crawlers │ │
│ │ - Filter: Look for events with user agents containing "Googlebot", "Bingbot" │ │
│ │ - Response: Create allow rules for any legitimate crawlers being blocked │ │
│ │ │ │
│ │ 5.3 Analytics & Monitoring │ │
│ │ │ │
│ │ - Location: Analytics → Security │ │
│ │ - Action: Monitor for search engine crawler blocks │ │
│ │ - Set Up: Alerts for when known search engines are being challenged/blocked │ │
│ │ │ │
│ │ Phase 6: 2025-Specific Considerations │ │
│ │ │ │
│ │ 6.1 AI Bot Blocking (New Default) │ │
│ │ │ │
│ │ - Cloudflare Change: Now blocks AI crawlers by default │ │
│ │ - Impact: Ensure this doesn't affect legitimate search crawlers │ │
│ │ - Action: Verify separation between AI bots and search engine crawlers │ │
│ │ │ │
│ │ 6.2 Verified Bot Categories │ │
│ │ │ │
│ │ - Location: Security → Bots → Configure │ │
│ │ - Action: Enable "Search Engine Crawler" category │ │
│ │ - Ensure: All major search engines are in verified bot allowlist │ │
│ │ │ │
│ │ Phase 7: Site-Specific Recommendations │ │
│ │ │ │
│ │ 7.1 Kuyadoga.com (Next.js) │ │
│ │ │ │
│ │ - Current: Browser Integrity Check OFF ✅ │ │
│ │ - Additional: Create comprehensive "Allow Search Bots" rule │ │
│ │ - Monitor: Ensure Next.js SSR doesn't conflict with crawler access │ │
│ │ │ │
│ │ 7.2 Malla-Supply.com (Traditional) │ │
│ │ │ │
│ │ - Current: Browser Integrity Check ON ✅ │ │
│ │ - Maintain: Current settings since indexing is working │ │
│ │ - Add: Verified bot allow rule as additional safeguard │ │
│ │ │ │
│ │ Implementation Priority │ │
│ │ │ │
│ │ High Priority (Do Immediately) │ │
│ │ │ │
│ │ 1. Create "Allow Verified Bots" WAF custom rule │ │
│ │ 2. Verify Bot Fight Mode settings │ │
│ │ 3. Check Security Level settings │ │
│ │ 4. Review any IP/User Agent blocking rules │ │
│ │ │ │
│ │ Medium Priority (This Week) │ │
│ │ │ │
│ │ 1. Audit Page Rules for crawler impact │ │
│ │ 2. Set up Security Events monitoring │ │
│ │ 3. Review Rate Limiting rules │ │
│ │ 4. Check Country/Geographic restrictions │ │
│ │ │ │
│ │ Low Priority (Ongoing) │ │
│ │ │ │
│ │ 1. Monitor Security Events for crawler blocks │ │
│ │ 2. Regular review of WAF Managed Rules updates │ │
│ │ 3. Optimize challenge settings based on traffic patterns │ │
│ │ │ │
│ │ Success Metrics │ │
│ │ │ │
│ │ - No legitimate search engine crawlers in Security Events │ │
│ │ - Improved indexing rates in Google Search Console │ │
│ │ - Maintained security posture against malicious bots │ │
│ │ - Reduced false positive blocks of legitimate traffic │ │
│ │ │ │
│ │ This comprehensive audit ensures your Cloudflare configuration optimally balances security with search engine accessibility, protecting against the most common issues that prevent proper indexing while │ │
│ │ maintaining strong security against malicious threats.