What's the best practice for rendering a high number of entries with exp:channel:entries?

Question

I'm running into memory issues with a page that lists lots (hundreds, eventually 1000+) of little news items that consist of a title, date, and sometimes brief description. These are each channel entries, and I've taken the following steps to reduce memory usage and speed up page rendering:

• utilize the {exp:channel:entries} disable parameter (disable="categories|category_fields|member_data|pagination")
• cache the entire page output with CE Cache ({exp:ce_cache:it id="page" seconds="3600"})
• maintain a relatively high PHP memory limit (256MB)
• avoid rendering any more complex fields (Playa, Matrix, etc.), which are this case part of the entries but not used on the page

I also attempted to temporarily boost PHP's memory using ini_set('memory_limit', '1024M') before the exp:channel:entries call in the template, then ini_set('memory_limit', '256M') after, but this doesn't work. (Never tried this in a template, so I'm not surprised anyway.)

Are there any common practices or alternate approaches that would be appropriate here?

Or, is this just a nonsensical goal that's forcing a no-win outcome?

Answer 1

One thing worth looking at is how well the {exp:channel:entries} loop you describes performs in a template with nothing else--no embeds, plugins, etc., vs in the context of the actual page on your site. It's important to be sure that your general template / page architecture isn't putting too heavy a load on EE / your server.

In other words, in addition to what you've done above, the caching you do on other elements on the page, and across your whole site, may be very relevant. In general, I've found that caching more, and for longer times, across a whole site is a good way to leave space for queries / processes that need memory and processor time.

I've dealt with a bunch of EE memory limit issues over the years, and I rely heavily on CE Cache these days. Where possible, I especially like using the static cache in the mode where it totally bypasses EE. But, beyond caching, I've had to do some combination of the following:

• add more memory -- the obvious!!!
• use a CRON job to generate close-to-final-format results in the database, and pull those into the template using a simple {exp:query}
• ditch using EE channel or query tags, and do SQL queries directly from PHP (because, it's less overhead and you can really optimize your SQL)

Answer 2

Matt, a couple different approaches for you.

I assume this is a page hit frequently.

1) use pagination in your channel entries tag to reduce the number of results displayed

2) cache the page using template preferences

3) use stash to cache the channel entries fields you need. For example here is a rough framework for the approach - ( hint - you need to add some paramaters to these tags)

{exp:stash:set_list [add params] save="yes" refresh="1440"}
{exp:channel:entries [add params] disable=[everthing except pagination]}
{stash:title}{title}{/stash:title}
{stash:date}{expiration_date}{/stash:date}
...And so on
{/exp:channel:entries}
{/exp:stash:set}


Then later in the page use get_list to display results in your markup.

4) Advanced: create your own plugin to return results, the use then stash to cache it. This approach worked us on one landing page because we needed to display up to 100 channel entries plus channel_images for each ( extra queries) plus matrix (extra queries). Using the channel entries tag, the number of queries increased relative to the number of results. In the end 200+ queries and huge memory. Now, about 33 queries (doesnt increase relative to num of results), memory about 4mb and speed about 0.43 seconds.

Do you need pagination? If so, copy the query module, replace $sql with your own query and away you go.

5) use automatee to refresh your cache results

6) limit page to only one or two embeds