Is it efficient enought for a large corpora built with Drupal? about crow_backend HOT 1 CLOSED

Thanks for asking about this. The production version of this application currently provides searchable indexing for around 18,000 individual texts (stored as Drupal nodes), and is able to provide results matching search queries in usually less than one second.

You are right to be cautious about trying to create a "big data" style application using Drupal. This is because Drupal, like most standard content management systems, uses a relational database structure to organize its content. Every field on a Drupal node, for example, is stored in a separate database table, so requesting a single node of data will require multiple SQL query joins.

I debated creating a custom entity in Drupal with a flatter database structure -- you can define a series of primary fields on an entity that would all be stored within a single table -- but so far, the traditional relational database has worked sufficiently.

That said, there is a fair amount of custom PHP code around doing the queries for the search results, specifically designed around handling big data. The main corpus search, for example, is not configure through a standard Drupal "View," due to that being less performant than a custom SQL query.

In addition the design relies on multiple levels of caching, as well as the fact that it returns its data as JSON content that is then presented through a separate front-end application (see https://github.com/writecrow/crow_frontend), both of which contribute to the perceived speed for the end user.

If you have additional questions about specific code, let me know!

from crow_backend.