Operating Modes
The connector can operate in the following modes:
- Streaming (Real-Time) Mode. The connector retrieves data that is streamed from Gnip in real-time. The connector can only retrieve entries that are streamed while the
synchronize
action is running. Any entries that are streamed when thesynchronize
action is not running are not retrieved. In this mode, thesynchronize
action does not stop unless you set theTaskMaxDuration
configuration parameter, run theStopFetch
action, or stop the connector. -
Historical Mode. The connector retrieves historical data from Gnip. In this mode, the
synchronize
action finishes when all of the data you request has been ingested. Historical jobs can take a long time to complete, but you can stop them by sending aStopFetch
action to the connector, and resume them by sending anothersynchronize
action.You can choose the time period to retrieve data for by specifying either fixed or relative start and end dates. Relative dates are relative to the time when the
synchronize
action starts. For example, each time you synchronize with Gnip you could choose to retrieve historical data for the last week. Gnip Connector does not send remove commands to your ingestion server for entries that no longer fall within the time period, but if you want your IDOL index to contain only entries from the last 7 days you can configure IDOL Server to expire documents.