Knowledge Discovery 25.1
Knowledge Discovery Expert

# Import Tasks

Import Tasks are processing tasks that are performed on documents by CFS, before the documents are indexed into the Content component. Import Tasks enable you to manipulate and enrich the documents that are created by CFS.

CFS includes Import Tasks that meet common processing requirements. For example, there are Import Tasks to filter advertisements out of HTML files, or divide document content into shorter sections.

## Write documents to disk

You can use the `IdxWriter` and `XmlWriter` tasks to write documents to disk in Knowledge Discovery IDX or XML format. This allows you to view the information that is being indexed into the Content component, so that you can check the information is being indexed as you expected. If necessary, you can then use other import tasks or custom Lua scripts to manipulate and enrich the information.

The `CsvWriter` and `JsonWriter` tasks write documents to disk in CSV or JSON format. You can also use the `SqlWriter` task to write document metadata and content to disk in the form of SQL "insert" statements, so that you can insert the information from the documents into a database.

## Manipulate and enrich documents

You can use import tasks to enrich documents, without needing to write custom scripts. For example, you can:

- use the `HtmlExtraction` import task to extract the meaningful content from HTML, and discard advertisements, headers, and sidebars.

- use the `Sectioner` import task to divide document content into shorter sections. Dividing a document can result in more relevant query results, because Content can return a specific part of a document in response to a query.

- use the `Eduction` import task to run Eduction.

- use the `MediaServerAnalysis` import task to run analysis on image, video, and audio files. You can run analysis tasks such as optical character recognition (OCR), object detection, and face recognition. You can extract speech from audio and video files, and write a transcription of the speech to the document content, which Content can use for retrieval, clustering, and other operations.

## Validate and reject documents

You can use import tasks to reject documents that you do not want to index into Content. For example, the `BadFilesFilter` task rejects documents that do not contain valid content. When a document is rejected, it is not processed further and is not indexed into Content. However, you can index the document into a Content that has been configured to handle failed documents.

## Run a Lua Script

The Lua task runs a Lua Script. Lua is an embedded scripting language that you can use to manipulate documents and define custom processing rules. CFS includes Lua functions for manipulating documents and running other tasks. For example, you can add, modify, or remove fields and their values.

## Configure Import Tasks

Import tasks are configured in the `[ImportTasks]` section of the CFS configuration file.

You can run Import Tasks before or after documents are processed by File Content Extraction. Pre import tasks run before File Content Extraction processing. Post Import tasks run after File Content Extraction processing.

- Pre Import Tasks are often used to control processing.

- Post Import Tasks are often used to write a document to disk (in either IDX or XML format). You then have a backup copy of the content that is indexed into your Content component, and you can see how the data is sent to Content.

Copyright 2024 Open Text
Version 25.1 | Last updated January 2025

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: Knowledge Discovery 25.1

Topic Title: Import Tasks