User-agent: * Disallow: /company-profile/rating/ Disallow: /product-category/list-basic-products* Disallow: /product-profile/rating/ Disallow: /product-profile/list Disallow: /product-profile/index Disallow: /company-profile/list Disallow: /assets/ Disallow: /compare/ Disallow: /review/ Disallow: /company-profile/update-company-profile User-agent: AhrefsBot User-agent: SemrushBot Disallow: /