Knowledge Discovery 25.1
Knowledge Discovery Expert

# Expand Knowledge Discovery Installations

A single Content component allows you to set up a small index and query your documents. However, most real-world uses of Knowledge Discovery outgrow this scenario fairly rapidly. Keeping all documents in a single instance of Content becomes unsustainable in a growing system.

- As the amount of data increases, so does the volume of the index on disk. If you keep on indexing data, it might use all the available disk space on the machine. In practice, Content requires a certain amount of free space to continue normal operations, so it automatically stops indexing more documents before it reaches this point.

- In normal operation, Content holds speed-critical parts of the index in memory. As the index size increases, so does the amount of physical memory required. At this point, the process swaps out its virtual memory (for example, to a page file), which has a corresponding impact on performance. As the process memory usage increases, the amount of free memory available to the operating system to cache frequently-accessed data from disk also decreases, with a further impact on performance.

- Query time tends to increase as the number of documents and the number of term occurrences (the volume of data to search) increases. When you add documents, Content must merge new data with the existing index, so indexing performance can also decrease as the index grows.

## Distributed Knowledge Discovery

You can distribute your index by running multiple instances of the Content component, each of which indexes a different subset of documents. Each instance is a self-contained index, containing a fraction of the complete body of documents. At query time, results are collated to combine search across the combined set of indexed documents.

The individual indexes are generally spread across multiple physical servers, allowing the disk, memory, and processor footprint of the combined index to expand beyond the limitations of a single machine. Even on a single machine, it is common practice to spread documents across several smaller copies of Content, rather than a single large instance. This approach can still improve performance, because the individual instances can work in parallel on the sub-indexes, bringing back results more quickly than a single large Content instance.

In addition to splitting data between servers, a distributed index can hold multiple copies of a document in different servers, for purposes of load balancing, failover, or disaster recovery.

Next: Scale Knowledge Discovery: Distributed Architecture

Related Topics

- Memory Usage in the Content component

- Distribution

- Improve Content component Performance

Copyright 2024 Open Text
Version 25.1 | Last updated January 2025

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: Knowledge Discovery 25.1

Topic Title: Expand Knowledge Discovery Installations