Open topic with navigation

# Preparing the Compute Engine VM instance

to come from the Fusion Database team. Mentions of azure in this procedure will be changed once the process is verified

# Installing the Database in Google Cloud

This section provides information about installing the ArcSight Database in Google Cloud.

- Understanding Azure Blob Storage

- Preparing the Azure Virtual Machine
- Installing Prerequisites
- Configuring and Installing the Database Server

## Understanding Azure Blob Storage

The database uses a single communal storage location for all data and for the catalog (metadata). Communal storage is the database's centralized storage location, shared among the database nodes. This mode supports communal storage in Azure Blob Storage, which must be set up by your cloud administrator before you can install the database. To install the database with communal storage as Azure Blob Storage, you need the following:

1. An Azure storage account name with a valid Azure subscription.

2. An Azure storage container.

3. An Azure account key.

For more information see, Azure Portal.

## Preparing the Azure Virtual Machine

This section describes how to prepare the Azure virtual machine for database installation.

1. Log in to the Azure portal.
2. Select an active Azure subscription.
3. Click Create a Resource or select an existing resource.
4. In the search box (case-insensitive), enter RHEL-8.6 minimal.
5. Click create.
6. In the Virtual machine name field, enter a name for the virtual machine.
7. In the Image drop-down list, select supported image.
8. In the Size drop-down list, select the VM that will be accomplishing the database requirements. For example, D8s_v3.
9. Select the appropriate Authentication type, and enter the details.
10. In the Select inbound ports drop-down list, be sure SSH (22) is selected.
11. In the Public inbound ports area, select the Allow selected ports option.
12. Click Next: Disks.
13. From the Disks tab and create a new disk of minimum size 256 GB.
14. For Data disks and Host caching select Read/write.
15. Select the Networking and your Virtual network.
16. Create a subnet for the database and select it in the subnet drop down.
17. Click Review + create to review and create a VM ; wait to complete deployment.
18. Click Go to your resource and launch the new VM.
19. In the left navigation panel, click Overview, find DNS name, and click the associated link.
20. Set Assignment to static and enter the DNS name label.
21. Click Save.
22. In the left navigation pane, click Networking.
23. Select your Network Interface.
24. From the left menu, select IP configuration.
25. Navigate to the IP Forwarding Field, and select Enabled.
26. Click Save.
27. SSH to your VM with it's public IP address.
28. Log in with your VM user, then become root.
29. Change your root password.
30. Create a folder for Vertica by running the command:

    mkdir /opt/vertica

31. Create partitions on the datadisk.
    Vertica requires a minimum 2 GB swap partition irrespective of the amount of RAM installed. The remaining part of the disk, you can disk format ext4 and mount to /opt/vertica.

    [root@vertica1 ~]# fdisk /dev/sdc
    Welcome to fdisk (util-linux 2.23.2)

    Changes will remain in memory only, until you decide to write them.
    Be careful before using the write command.

    Device does not contain a recognized partition table
    Building a new DOS disklabel with disk identifier 0x2959fe99.

    The device presents a logical sector size that is smaller than
    the physical sector size. Aligning to a physical sector (or optimal
    I/O) size boundary is recommended, or performance may be impacted.

    Command (m for help): n
    Partition type:
    p   primary (0 primary, 0 extended, 4 free)
    e   extended
    Select (default p):
    Using default response p
    Partition number (1-4, default 1):
    First sector (2048-536870911, default 2048):
    Using default value 2048
    Last sector, +sectors or +size{K,M,G} (2048-536870911, default 536870911): ${sizePartitionForSwap}
    Partition 1 of type Linux and of size 4 GiB is set

    Command (m for help): n
    Partition type:
    p   primary (1 primary, 0 extended, 3 free)
    e   extended
    Select (default p):
    Using default response p
    Partition number (2-4, default 2):
    First sector (4196352-536870911, default 4196352):
    Using default value 4196352
    Last sector, +sectors or +size{K,M,G} (4196352-536870911, default 536870911):
    Using default value 536870911
    Partition 2 of type Linux and of size 254 GiB is set

    Command (m for help): p

    Disk /dev/sdc: 274.9 GB, 274877906944 bytes, 536870912 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 4096 bytes
    I/O size (minimum/optimal): 4096 bytes / 4096 bytes
    Disk label type: dos
    Disk identifier: 0xcca9a285

    Device Boot      Start         End      Blocks   Id  System
    /dev/sdc1         2048     2097152     1047552+  83  Linux
    /dev/sdc2      2099200   536870911   267385856   83  Linux

    Command (m for help): w
    The partition table has been altered!

    Calling ioctl() to re-read partition table.
    Syncing disks.

32. Create a swap by running the command:

    mkswap /dev/sdc1

33. Activate the swap by running the command:

    swapon /dev/sdc1

34. To format rest of disk to ext4, run the command:

    mkfs.ext4 /dev/sdc2

35. Get the last created UUID of disks from the output by running the command:

    blkid

36. For swap (/dev/sdc1) and for rest disk (/dev/sdc2) take note of the given UUID values, modify the /etc/fstab file, and add the following lines by replacing the UUIDs:

    UUID=<UUID sdc1>    none	  swap	sw	  0 0
    UUID=<UUID sdc2>    /opt/vertica  ext4	 defaults  0 0

37. Mount all by running the command:

    mount -a

38. Check if `/opt/vertica` is assigned by running the command:

    df -h

39. Enable the swap by running the command:

    swapon -a

40. Check the swap size by running the command:

    free -h

41. Repeat these steps for each expected database node.

## Installing Prerequisites

This section describes how to install the prerequisites necessary to install the Azure database.

1. Configure passwordless communication from the node1 server to all of the node servers in the cluster.

1. On the node1 server, run the ssh-keygen command:

       ssh-keygen -t rsa

2. Copy the key from node1 to all of the nodes, including node1, using the node IP address:

       ssh-copy-id -i ~/.ssh/id_rsa.pub root@$node1,root@$node2,root@$node3

2. Set up and activate `/etc/rc.local` by running the following command:

    #!/bin/sh
    function drive {
    block_device=`realpath $(df $1 | grep '^/' | cut -d' ' -f1)`
    partition=$(echo $block_device | sed -e "s#/dev/##")
    if [[ $partition == dm-* ]]; then
    echo $partition
    else
    echo $partition | cut -c1-3
    fi
    }
    cat > /etc/rc.local << EOF
    #!/bin/sh
    touch /var/lock/subsys/local
    /sbin/blockdev --setra 2048 /dev/$(drive /)
    /sbin/blockdev --setra 2048 /dev/$(drive /opt)
    echo deadline > /sys/block/$(drive /)/queue/scheduler
    echo deadline > /sys/block/$(drive /opt)/queue/scheduler
    echo never > /sys/kernel/mm/transparent_hugepage/enabled
    tuned-adm profile throughput-performance
    EOF
    chmod 755 /etc/rc.local
    /etc/rc.local

3. Add the following parameters to `/etc/sysctl.conf`.

Parameter | Description
`net.core.somaxconn = 1024` | Increases the number of incoming connections
`net.core.wmem_max = 16777216` | Sets the send socket buffer maximum size in bytes
`net.core.rmem_max = 16777216` | Sets the receive socket buffer maximum size in bytes
`net.core.wmem_default = 262144` | Sets the receive socket buffer default size in bytes
`net.core.rmem_default = 262144` | Controls the default size of receive buffers used by sockets
`net.core.netdev_max_backlog = 100000` | Increase the length of the network interface input queue
`net.ipv4.tcp_mem = 16777216 16777216 16777216`
`net.ipv4.tcp_wmem = 8192 262144 8388608`
`net.ipv4.tcp_rmem = 8192 262144 8388608`
`net.ipv4.udp_mem = 16777216 16777216 16777216`
`net.ipv4.udp_rmem_min = 16384`
`net.ipv4.udp_wmem_min = 16384`
`vm.swappiness = 0` | Defines the amount and frequency at which the kernel copies RAM contents to a swap space For more information, see Check for Swappiness in the ArcSight Database Guide.

4. Next, run the following command:

    sysctl -p

5. To disable the firewall WARN (N0010), use `iptables`:

   `iptables -F`

   `iptables -t nat -F`

   `iptables -t mangle -F`

   `iptables -X`

   `systemctl mask firewalld`

   `systemctl disable firewalld`

   `systemctl stop firewalld`

   The database requires several ports to be open on the local network. Micro Focus does not recommend that you place a firewall between nodes (all nodes should be behind a firewall), but if you must use a firewall between nodes, ensure that all the database ports are available (see Firewall ports in Technical Requirements for ArcSight Platform 23.3). For more information, see Firewall Considerations in ArcSight Database Guide.
6. Set SELinux to permissive mode in `/etc/selinux/config`.

       SELINUX=permissive

   For more information, see SELinux Configuration in ArcSight Database Guide.

7. In `/etc/default/grub`, append line `GRUB_CMDLINE_LINUX` with `intel_idle.max_cstate=0 processor.max_cstate=1`.

   For example:

       GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet intel_idle.max_cstate=0 processor.max_cstate=1 intel_pstate=disable"

   Execute the following command:

       grub2-mkconfig -o /boot/grub2/grub.cfg

8. If you have a high concurrency workload and if the database is CPU bound, reboot the virtual machine by running the following command:

    sudo sysctl -w net.core.netdev_max_backlog=2000

9. Reboot for your changes to take effect.
10. For RHEL, you must run RHEL using the following command:

    dnf install libnsl

11. Install the packages by running the following command:

    yum install -y java-1.8.0-openjdk gdb mcelog sysstat dialog chrony tzdata wget

12. Modify the /etc/bashrc by running the following command:

    export VERTICA_FAILURE_THRESHOLD=FAIL

13. Apply the changes by running the following command:

    source ~/.bashrc

14. Repeat these steps for each expected database node.

## Configuring and Installing the Database Server

This section describes how to configure and install the Azure database.

Before installing the database, ensure that you estimate the storage needed for the incoming EPS (event per second) and event size, and also evaluate the retention policy accordingly.

1. On the Database cluster node1 server, create a folder for the database installer.

   For example:

       mkdir /opt/arcsight-db-tools

`/opt/arcsight-db-tools` should not be under `/root` or `/opt/vertica`.

2. Download `arcsight-platform-cloud-installer-23.3.x.zip` and unzip. The directory to where you unzip it will be referred to as {`Unzipped-arcsight-platform-cloud-installer-folder}`.

3. Copy the Database binaries from `{Unzipped-arcsight-platform-cloud-installer-folder/database}` to your Database cluster node1 server, at the `/opt/arcsight-db-tools` directory.

4. Edit the `config/db_user.properties` file and add all database node IPs to the hosts property.

   Property | Description
   --- | ---
   `hosts` | A comma separated list of the database servers in IPv4 format (for example, 1.1.1.1,1.1.1.2,1.1.1.3). If it is necessary to construct the cluster, avoid using local loopback (localhost, 127.0.0.1, etc.).

5. Install the database.

       ./db_installer install

6. When prompted, create the database administrator user.

   The database administrator user account is used during database deployment, configuration, upgrade, and debugging. For security reasons, the platform deployed capabilities will not ask you for the credentials for this user.

       ---------------------------------------
       Please specify a username for [ DB Admin ] user:
       dbadmin

       ---------------------------------------
       Please specify a password for [ DB Admin ] user:
       ********
       Re-enter password:
       ********

   For a list of options that you can specify when installing the database, see Understanding the Database Installer Options.

7. Specify the shard count. The default shard count is 3 for single-node or 18 for multi-node to allow for scalability.

   - Single-node:

         # ========================================
         # STEP 1: Specify Database Shard Count for Eon Mode
         Do you plan to keep the database cluster to a single node in the future?
         If yes, the database will be optimized for performance on a single node by setting the default shard count to 3.
         Shard Count [3]:
         Confirm shard count [3]?(y/n):y
         Check memory size, 48GB required for single node installation with shard count > 3.
         PASS: Single node installation for shard count: 3

   - Multi-node:

         # ========================================
         # STEP 1: Specify Database Shard Count for Eon Mode
         Recommended shard count for multi node database deployment is 18.
         Shard Count [18]:
         Confirm shard count [18]?(y/n):y

8. Set up the communal storage type for Azure Blob Storage when prompted. For example:

       # ========================================
       # STEP 2: Specify communal storage details
       Supported communal storage types -
       1) S3
       2) Azure Blob Storage
       Choose a communal storage type from the above (1/2):2
       Specify account name: <your Azure account name>
       Do you want to use managed identity to authenticate with Azure storage container(y/n):n
       Specify account key:
       Is TLS enabled(y/n):y
       Specify Azure container for communal storage: <your Azure container name>
       Specify the folder under container for communal storage if applicable:
       Communal storage url is: https://<your account name>.blob.core.windows.net/mycontainer/myblob

9. Create the schema.

       ./db_installer create-schema

10. When prompted, create the following users:

    - App admin user: A regular database user granted elevated permissions for performing operations on the database to manage the database, schema, and resource pools. The credentials for this user will need to be provided later in the CDF Management Portal when you are deploying capabilities.

    - Search user: A regular database user with permissions restricted to event search operations. The credentials for this user will need to be provided later in the CDF Management Portal when you are deploying capabilities.

11. Monitor your database cluster status constantly. For more information, see Database Cluster Node Status and Monitoring the Database.

    - Database nodes status: Ensures all nodes are up

    - Database nodes storage status: Ensures storage is sufficient