Open topic with navigation

Configure IDOL > Languages > Tokenization > Determine the Tokens to Index > Numeric and Alphanumeric Terms

# Numeric and Alphanumeric Terms

When you configure your IDOL Server index, you must consider the terms that you want to search for, and the terms that you want to index. Numeric and alphanumeric terms can take up a large proportion of an index, so it is important to consider what you need to search for, and configure your index appropriately.

TIP:

You can view information on the number of numeric and alphanumeric terms in your data index on the Terms tab on the Performance page in the Monitor section of IDOL Admin.

## Configuration Parameters

The following table outlines some of the configuration parameters that are useful to consider. For more details, refer to the IDOL Server Reference.

Parameter | Section | Description
--- | --- | ---
`SplitNumbers` | `[Server]` | Split numeric and alphanumeric terms and store the chunks as separate terms.
`IndexNumbers1TruncateLength` | `[Server]` | The maximum length of a purely numeric terms. Longer numeric terms are truncated before indexing.
`IndexNumbers2TruncateLength` | `[Server]` | The maximum length of alphanumeric terms. Longer alphanumeric terms are truncated before indexing.
`NumericTermChunkSize` | `[Server]` | The maximum number of characters per chunk when splitting a purely numeric term.
`AlphaNumericTermChunkSize` | `[Server]` | The maximum number of characters per chunk when splitting an alphanumeric term.
`IndexNumbersMaxValue` | `[Server]` | The maximum value of a numeric term that you want to index.
`UnstemmedIndexNumbers` | `[Server]` | Whether to add numeric terms to the unstemmed index.
`UnstemmedIndexNumbers0MaxLength` | `[Server]` | The maximum length of non-numeric values to add to the unstemmed index.
`UnstemmedIndexNumbers1MaxLength` | `[Server]` | The maximum length of purely numeric values to add to the unstemmed index.
`UnstemmedIndexNumbers2MaxLength` | `[Server]` | The maximum length of alphanumeric values to add to the unstemmed index.
`IndexNumbers` | `[MyLanguage]` | The method to use to handle numbers during indexing. See IndexNumbers.
`IndexNumbers` | `[MyFieldProperty]` | Restricts the per-language `IndexNumbers` setting for a specific field or fields with the `IndexNumbersType` property.
`IndexNumbersType` | `[MyFieldProperty]` | A field property that specifies that certain fields must use the per-field `IndexNumbers` setting.
`IndexNumbers1MaxLength` | `[MyFieldProperty]` | The maximum length of purely numeric terms to index for fields with this property.
`IndexNumbers2MaxLength` | `[MyFieldProperty]` | The maximum length of alphanumeric terms to index for fields with this property.

NOTE:

Truncation applies globally across all fields to ensure consistency at query time.

## Index Process

At index time, IDOL Server runs the following process to determine how to index numeric and alphanumeric terms:

1. IDOL determines whether the term is non-numeric, numeric, or alphanumeric. According to your per-language `IndexNumbers` configuration setting, it discards any terms that you do not want to index.

2. IDOL discards pure numeric terms that are greater than `IndexNumbersMaxValue`, and uses the field property settings to discard terms longer than the relevant `IndexNumbersNMaxLength`, and terms that do not match the `IndexNumbers` setting for the field.

3. IDOL truncates numeric and alphanumeric terms according to the values of the `IndexNumbersNTruncateLength` parameters.

4. If `SplitNumbers` is set to `True`, IDOL splits numeric terms into chunks of `NumericTermChunkSize`, and alphanumeric terms into chunks of `AlphaNumericTermChunkSize`.

   TIP:

   You can set `NumericTermChunkSize` or `AlphaNumericTermChunkSize` to `-1`, which means that it does not split terms for that type. For example, you might want to split numbers only for purely numeric terms, in which case you can set `AlphaNumericTermChunkSize` to `-1`.

5. IDOL adds normal terms to the unstemmed index, and indexed into the dynterm index. In the dynterm index, it stems the terms and then truncates terms to `TermSize`. Numeric and alphanumeric terms are never stemmed.

   - Terms that have been chunked by `SplitNumbers` do not get indexed into the unstemmed index (even when they are short enough to be a single chunk).

   - If you have set `NumericTermChunkSize` or `AlphaNumericTermChunkSize` to -1, the associated terms are indexed into the unstemmed index.

   - Terms longer than `UnstemmedIndexNumbersNMaxLength` are not added to the unstemmed index.

## Queries

When you run queries for numeric and alphanumeric terms, the results depend on your configuration. In general:

- you cannot search for any term that is discarded at index time (similar to stop words). For example, if you set `IndexNumbers` to `0`, you cannot search for any numeric or alphanumeric terms.

- you cannot use wildcards to search for any term that is not added to the unstemmed index. For example, if you set `SplitNumbers` to `True`, numeric terms are split into chunks, and are not added to the unstemmed index, so you cannot use wildcards to search for them. You can search for the exact term as normal.

  NOTE:

  The `TermGetInfo` action returns information about the term chunks that an alphanumeric term is split into. However, when you send a query, you can search only for the number in its entirety, and not for the individual term chunks.

When you decide how you want to configure your IDOL Server to handle numeric and alphanumeric data, you must consider whether you need to search for these values at all, and whether you want to use wildcards to search for them.

In many cases, you do not need to use wildcards to search for numbers. For example, you might have invoice numbers that do not have any special significance except for the order. You might want to be able to search for a specific invoice, but you usually know the exact number that you want to find. If you never want to use wildcards to search for values, you can use `UnstemmedIndexNumbers` to prevent IDOL Server from storing the unstemmed terms.

If you index spreadsheets, it might add lots of terms to the unstemmed index that you never need to search for with wildcards. In this case, you might want to disable unstemmed indexing of numbers, and you might also want to use `SplitNumbers` to reduce the total number of numeric terms that IDOL indexes.

For numbers that you search for regularly, you might want to use Eduction to extract the number to a field. You can then use FieldText operators to search for the numbers. For example, if you want to search for ranges of invoice numbers, you can use the `RANGE` operator. You can use the NumericType field property for the field to optimize these operations.

## Example

The following example shows the configuration options for a particular scenario, and the results of indexing various numeric terms into an IDOL Server with this configuration.

    [Server]
    SplitNumbers=True
    IndexNumbers1TruncateLength=12
    IndexNumbers2TruncateLength=0
    NumericTermChunkSize=7
    AlphaNumericTermChunkSize=-1
    IndexNumbersMaxValue=0              // (explicit default)
    UnstemmedIndexNumbers0MaxLength=-1  // (explicit default)
    UnstemmedIndexNumbers1MaxLength=-1  // (explicit default)
    UnstemmedIndexNumbers2MaxLength=-1  // (explicit default)

    [MyLanguage]
    IndexNumbers=1

    [ No field-specific overrides ]

- `a123456789`. Indexed without modification, returnable in a wildcard search.

- `1234`. Indexed as a single SplitNumber chunk `12341Z`, not returnable in a wildcard search.

- `12345678`. Indexed as two SplitNumber chunks: `12Z`, `23456781`, not returnable in a wildcard search.

- `123456789012`. Indexed as two SplitNumber chunks: `123452Z`, `67890121`, not returnable in a wildcard search.

- `123456789012000000`. Truncated and indexed as two SplitNumber chunks: `123452Z`, `67890121`, not returnable in a wildcard search.

- `A123456789B123456789C123456789D12345678`. Indexed without modification, returnable in a wildcard search.

- `A123456789B123456789C123456789D123456789E123456789`. Truncated and indexed as `A123456789B123456789C123456789D123456789` (that is, based on `TermSize=40`), returnable in some wildcard searches (for example, `A*` but not `*789`).

© 2017 Hewlett Packard Enterprise Development LP | Send documentation feedback to HPE Close Send documentation feedback To open the configured email client on this computer, open an email window. Otherwise, copy the information below to a web mail client, and send this email to AutonomyTPFeedback@hpe.com. We appreciate your feedback! Product: Topic Title: Help Topic: Feedback:

_HP_HTML5_bannerTitle.htm