Redesigning or optimizing your category or section pages? There’s one element that every UX, Design, SEO, and Development teams can never seem to agree on.
Infinite scroll is so cool but load more is better for page performance. And pagination? OMG, what is this, 1995?! Believe me, I heard it all! But as my colleague, Eddie Alvarez tells me — There should always be a GOOD reason for every design choice.
What better to help make that design choice than technical SEO? Because if bots — whether traditional search bots like GoogleBot, BingBot or AI bots like ChatGPT or Anthropic — every website needs to make their content accessible, indexable, and findable by your potential customers, right?
After hearing this question enough times I finally spent the time to do the research, and tested various methods of load more, infinite scroll, and standard pagination to identify which methods produced the deepest crawl depths. Here’s what we found:
Load More
Load more was without a doubt the worst option for crawling.
Why? The bots are not humans. They do not interact with your website. So any JavaScript based functions, which hides the anchor tag’s URL from the code until after a human user clicks the button, will never be seen by the bot.
Therefore, when they come to using a “load more” button, crawling essentially stops at the end of entries for a given category page.
Infinite Scroll
When we did our testing, we found that bots would crawl further but that didn’t happen until we got the code exactly right and it took a number of sprints before that happened.
Our initial implementation loaded results very slowly and that process only began when the very end of the page was hit. From a user perspective, it didn’t feel at all like infinite scroll should feel and didn’t improve crawl depth whatsoever. At this stage of testing, this form of infinite scroll was performing pretty much the same as load more.
Thankfully, I didn’t give up there! It took a few development cycles but we finally started getting the visible and crawl results we were looking for!
What changeD?
A number of things — and it was about finding exactly the right balance for our site:
- SMALLER BATCH OF INITIAL POSTS: Initially, this seemed counter productive but really helped all the category page’s Core Web Vitals metrics, specifically LCP (Largest Contentful Paint) and INP (Interaction To Next Paint).
- EARLIER LOADING: We had to begin loading the new results sets significantly earlier in the process. The best results occurred when we started loading the first batch of posts at the end of the first viewport.
- LOAD IN SMALLER BATCHES OF POSTS : The smaller batches of more posts really improved the time to load the new result set into the page. Our sweet spot was around 21 posts per load.
It may take some time to tweak but well worth it in the long run! Here’s more info getting the best infinite scroll bot performance from Google’s Search Central.
Now for the bad news…the biggest downfall to using infinite scroll was just like users, the bot seemed to get bored at some point after sorting through results and eventually stopped crawling new URLs. The odd thing is this seems to vary from category to category but my best guess is that it’s based on publishing frequency to the various category pages.
If your site isn’t that large, the problem is far less but for sites that produce over 100K pages, the bot will never be able to crawl as deep as you need it to.
Standard Pagination
I know pagination is not sexy or cool, but to get the best crawl metrics, it’s is still the very best option as bots will follow every single anchor tag URL until there’s nothing left.
You know what they say — “If it ain’t broke, don’t fix it”. In the case of technical SEO this is, and likely always will be, the best way to go.
Teams still can’t agree?
So what’s the best past forward? Seems clear to me but I get the appeal of newer, fresher elements. And when it comes to getting UX, Designers, and Devs to agree, there’s always a “but” — some reason why one option can’t or won’t work.
But we’re SEO’s, right? We thrive at finding a middle ground — a solution that will work for bots and users. So why not try a hybrid approach? UX really wants infinite scroll but SEO is concerned about all posts getting crawled? Do them both!
What used to be known as Dynamic Rendering or serving one page to the bots and another to users as a workaround for using JavaScript based functions has changed a bit, but the concept is still relatively the same. Google now recommends using either server-side rendering, static rendering, or hydration as a solution.
Unless you have a former developer on your SEO team who knows your stack intimately, I suggest sharing these resources with your dev team and let them tell you what’s possible on your site.
Or, you’re always welcome to reach out to Transformation Labs for help…
