Copy Logger Data to SaaS

This process copies the data files and corresponding metadata for the selected archives to the Amazon S3 bucket.

  1. Perform Steps 1–3 of the Running the Archive Migration Tool procedure.

  2. Select Archives Available from the initial menu:

    What do you want to do?
    [ ] Manage S3 Bucket
    [ x ] Archives Available
    [ ] Logger Information
    [ ] Generator ID
    [ ] Generate Catalog
    [ ] Exit

  3. Select one or several of the Storage groups listed at the top of the menu.

  4. Select Copy.

    For example:

    [ x ] Internal Event Storage Group (Total: xGb, Ava: xGb, Pen: xGb, Done: xGb, Fail: xGb)
    What do you want to do?
    [ x ] Copy
    [ ] Schedule
    [ ] More details
    [ ] Back

  5. (Conditional) To choose archives from specific years, months or days, select More details.

    For more information about the levels of granularity to choose files from specific dates, see Viewing the Archives Contained in a Logger.

    Note: If the Generator ID has not been configured, and it's not enabled by default in Logger, checking an option from the More Details menu will produce the following prompt:

    To activate the migration process, check the Generator ID

    To configure the option, which is required to initiate migrations, you must go back to the main menu, as detailed in Adding a Unique ID to Migrated Events.

  6. After you select the archives and Copy, the system executes several verification steps to assess the availability and completeness of the archives to be migrated:

    • First, the existence and accessibility of the given path is verified. If either of these things cannot be verified, the tool will mark the “migration_status” as failed, and the “archive_status” as Not Accessed, and that archive will not be included in the Archive Catalog.

    • If the previous step completes successfully, the size and checksum of the files accessed inside the path is calculated (only for files with .xml, .csv and .dat extensions). The size and checksum values are then attached to the Archive Catalog.

      Note: Be aware that the global process of calculating the size and checksum for each file will take time, depending on size and file number.

    • Lastly, the system verifies the completeness of the archive. The archive size will indicate the number of datafiles (each 1 GB in size) that it should contain; an archive must have the same number of .csv and datafiles, and just one xml, so the tool would expect that if the archive size is x, the number of files contained in it will be 2x+1.

      If the above does not add up, the "archive_status" will be marked as Incomplete and this will be documented in the Archive Catalog as well.

      • If any errors occur during this process, the Archive Migration Tool will print messages on the screen to make you aware of them.

      To review the execution log, look for the logger_to_recon_archive_catalog_${loggerIPNoDots}.log file, saved under the same path where the script is.

      During execution, the loggerToReconArchiveCatalog.sh script will print error messages only. To view more on-screen detail, activate the verbose mode by adding the -v parameter when executing the script.

      Once the copy process is completed, an Archive Catalog file is generated. This file contains metadata information about the archives that have been copied so far, plus additional information about Logger, such as its storage groups and their retention.

      The Archive Catalog file is copied to the Amazon S3 bucket in a folder named:

      Bucket_Name/event-sync/logger-archives/Tenant_ID/Logger_IP_Without_Dots/

      The copied Logger archives' files will be available in folders such as:

      Bucket_Name/event-sync/logger-archives/Tenant_ID/Logger_IP_Without_Dots/Storage_Group_ID/YearMonthDay/

  7. (Conditional) To migrate additional archives from the current Logger, select Yes at the prompt that displays after the previous copy process completes:

    Do you want to migrate more archives?
    [ x ] Yes
    [ ] No

    The system will take you back to the steps of the Copy Logger Data to SaaS procedure.

  8. If no more files will be copied, you may exit the tool.

  9. Repeat this process for each Logger whose archived data you want to import.

  10. (Optional) Schedule the data migration rather than copying the files right now.

  11. (Optional) To repeat the copy procedure for files that have already been processed, see Reprocessing Copied Files.