Installing the Database in Azure

This section provides information about installing the ArcSight Database in Azure.

 

Preparing the Azure Virtual Machine

This section describes how to prepare the Azure virtual machine for database installation.

  1. Log in to the Azure portal.
  2. Select an active Azure subscription.
  3. Click Create a Resource or select an existing resource.
  4. In the search box (case-insensitive), enter Centos-based HPC.
  5. Click create.
  6. In the Virtual machine name field, enter a name for the virtual machine.
  7. In the Image drop-down list, select supported image.
  8. In the Size drop-down list, select the VM that will be accomplishing the database requirements. For example, D8s_v3.
  9. Select the appropriate Authentication type, and enter the details.
  10. In the Select inbound ports drop-down list, be sure SSH (22) is selected.
  11. In the Public inbound ports area, select the Allow selected ports option.
  12. Click Next: Disks.
  13. From the Disks tab and create a new disk of minimum size 256 GB.
  14. For Data disks and Host caching select Read/write.
  15. Select the Networking and your Virtual network.
  16. Create a subnet for the database and select it in the subnet drop down.
  17. Click Review + create to review and create a VM ; wait to complete deployment.
  18. Click Go to your resource and launch the new VM.
  19. In the left navigation panel, click Overview, find DNS name, and click the associated link.
  20. Set Assignment to static and enter the DNS name label.
  21. Click Save.
  22. In the left navigation pane, click Networking.
  23. Select your Network Interface.
  24. From the left menu, select IP configuration.
  25. Navigate to the IP Forwarding Field, and select Enabled.
  26. Click Save.
  27. SSH to your VM with it's public IP address.
  28. Log in with your VM user, then become root.
  29. Change your root password.
  30. Create a folder for Vertica by running the command:
    31. mkdir /opt/vertica
  31. Create partitions on the datadisk.
    Vertica requires a minimum 2 GB swap partition irrespective of the amount of RAM installed. The remaining part of the disk, you can disk format ext4 and mount to /opt/vertica.
    32. [root@vertica1 ~]# fdisk /dev/sdc
Welcome to fdisk (util-linux 2.23.2)

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
 
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x2959fe99.
 
The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.
 
Command (m for help): n
Partition type:
p   primary (0 primary, 0 extended, 4 free)
e   extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (2048-536870911, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-536870911, default 536870911): ${sizePartitionForSwap}
Partition 1 of type Linux and of size 4 GiB is set
 
Command (m for help): n
Partition type:
p   primary (1 primary, 0 extended, 3 free)
e   extended
Select (default p):
Using default response p
Partition number (2-4, default 2):
First sector (4196352-536870911, default 4196352):
Using default value 4196352
Last sector, +sectors or +size{K,M,G} (4196352-536870911, default 536870911):
Using default value 536870911
Partition 2 of type Linux and of size 254 GiB is set
 
Command (m for help): p
 
Disk /dev/sdc: 274.9 GB, 274877906944 bytes, 536870912 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: dos
Disk identifier: 0xcca9a285
 
Device Boot      Start         End      Blocks   Id  System
/dev/sdc1         2048     2097152     1047552+  83  Linux
/dev/sdc2      2099200   536870911   267385856   83  Linux
 
Command (m for help): w
The partition table has been altered!
 
Calling ioctl() to re-read partition table.
Syncing disks.
  32. Create a swap by running the command:
    33. mkswap /dev/sdc1
  33. Activate the swap by running the command:
    34. swapon /dev/sdc1
  34. To format rest of disk to ext4, run the command:
    35. mkfs.ext4 /dev/sdc2
  35. Get the last created UUID of disks from the output by running the command:
    36. blkid
  36. For swap (/dev/sdc1) and for rest disk (/dev/sdc2) take note of the given UUID values, modify the /etc/fstab file, and add the following lines by replacing the UUIDs:
    37. UUID=<UUID sdc1>    none	  swap	sw	  0 0
UUID=<UUID sdc2>    /opt/vertica  ext4	 defaults  0 0
  37. Mount all by running the command:
    38. mount -a
  38. Check if /opt/vertica is assigned by running the command:
    39. df -h
  39. Enable the swap by running the command:
    40. swapon -a
  40. Check the swap size by running the command:
    41. free -h
  41. Repeat these steps for each expected database node.

 

Installing Prerequisites

This section describes how to install the prerequisites necessary to install the Azure database.

  1. Configure passwordless communication from the node1 server to all of the node servers in the cluster.

    1. On the node1 server, run the ssh-keygen command:

      ssh-keygen -t rsa

    2. Copy the key from node1 to all of the nodes, including node1, using the node IP address:

      ssh-copy-id -i ~/.ssh/id_rsa.pub root@$node1,root@$node2,root@$node3
  2. Set up and activate /etc/rc.local by running the following command:
    3. #!/bin/sh
function drive {
block_device=`realpath $(df $1 | grep '^/' | cut -d' ' -f1)`
partition=$(echo $block_device | sed -e "s#/dev/##")
if [[ $partition == dm-* ]]; then
echo $partition
else
echo $partition | cut -c1-3
fi
}
cat > /etc/rc.local << EOF
#!/bin/sh
touch /var/lock/subsys/local
/sbin/blockdev --setra 2048 /dev/$(drive /)
/sbin/blockdev --setra 2048 /dev/$(drive /opt)
echo deadline > /sys/block/$(drive /)/queue/scheduler
echo deadline > /sys/block/$(drive /opt)/queue/scheduler
echo always > /sys/kernel/mm/transparent_hugepage/enabled
tuned-adm profile throughput-performance
EOF
chmod 755 /etc/rc.local
/etc/rc.local

  3. Add the following parameters to /etc/sysctl.conf.

    4. Parameter Description
    net.core.somaxconn = 1024 Increases the number of incoming connections
    net.core.wmem_max = 16777216 Sets the send socket buffer maximum size in bytes
    net.core.rmem_max = 16777216 Sets the receive socket buffer maximum size in bytes
    net.core.wmem_default = 262144 Sets the receive socket buffer default size in bytes
    net.core.rmem_default = 262144 Controls the default size of receive buffers used by sockets
    net.core.netdev_max_backlog = 100000 Increase the length of the network interface input queue
    net.ipv4.tcp_mem = 16777216 16777216 16777216
    net.ipv4.tcp_wmem = 8192 262144 8388608
    net.ipv4.tcp_rmem = 8192 262144 8388608
    net.ipv4.udp_mem = 16777216 16777216 16777216
    net.ipv4.udp_rmem_min = 16384
    net.ipv4.udp_wmem_min = 16384
    vm.swappiness = 1

    Defines the amount and frequency at which the kernel copies RAM contents to a swap space

    For more information, see Check for Swappiness.

  4. Next, run the following command:
    5. sysctl -p

  5. To disable the firewall WARN (N0010), use iptables: 

    iptables -F

    iptables -t nat -F

    iptables -t mangle -F

    iptables -X

    systemctl mask firewalld

    systemctl disable firewalld

    systemctl stop firewalld

    The database requires several ports to be open on the local network. Micro Focus does not recommend that you place a firewall between nodes (all nodes should be behind a firewall), but if you must use a firewall between nodes, ensure that all the database ports are available (see Firewall ports in Technical Requirements for ArcSight Platform). For more information, see Firewall Considerations.
  6. Set SELinux to permissive mode in /etc/selinux/config. 
    SELINUX=permissive

    For more information, see SELinux Configuration.

  7. In /etc/default/grub, append line GRUB_CMDLINE_LINUX with intel_idle.max_cstate=0 processor.max_cstate=1 by running the following command:

    grub2-mkconfig -o /boot/grub2/grub.cfg

    For example:

    GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet intel_idle.max_cstate=0 processor.max_cstate=1 intel_pstate=disable"

  8. Depending on your OS, RHEL/CentOS 8.x or CentOS 7.x, run the following command:
    9. echo always > /sys/kernel/mm/transparent_hugepage/enabled   #### 7.7 and 7.8 always 8.1 madvise
##? echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
##> echo no > /sys/kernel/mm/redhat_transparent_hugepage/khugepaged/defrag
### Changed: cpupower frequency-set --governor performance #### CentOS only, resolve WARN (S0140/S0141)
  9. Run the following commands:
    10. myroot=`df -h | grep '/$' | awk '{print $1}'`
myopt=`df -h | grep '/opt' | awk '{print $1}'`
  
echo deadline > /sys/block/sdb/queue/scheduler    #### Resolve FAIL (S0150)
/sbin/blockdev --setra 8192 $myopt                #### Resolve FAIL (S0020)
/sbin/blockdev --setra 2048 $myroot
echo always > /sys/kernel/mm/transparent_hugepage/enabled  
echo deadline > /sys/block/sda/queue/scheduler
tuned-adm profile throughput-performance #### CentOS only, resolve WARN (S0140/S0141)
  
/sbin/blockdev --setra 2048 /dev/$drive
echo deadline > /sys/block/$drive/queue/scheduler
echo always > /sys/kernel/mm/transparent_hugepage/enabled  
tuned-adm profile throughput-performance #### CentOS only, resolve WARN (S0140/S0141)
  10. If you have a high concurrency workload and if the database is CPU bound, reboot the virtual machine by running the following command:
    11. sudo sysctl -w net.core.netdev_max_backlog=2000
  11. Reboot for your changes to take effect.
  12. For RHEL/CentOS 8.x, you must run RHEL/CentOS 8.x using the following command:
    13. dnf install libnsl
  13. Install the packages by running the following command:
    14. yum install -y java-1.8.0-openjdk gdb mcelog sysstat dialog chrony tzdata wget
  14. Modify the /etc/bashrc by running the following command:
    15. export VERTICA_FAILURE_THRESHOLD=FAIL
  15. Apply the changes by running the following command:
    16. ~/.bashrc
  16. Repeat these steps for each expected database node.

 

Configuring and Installing the Database Server

This section describes how to configure and install the Azure database.

Before installing the database, ensure that you estimate the storage needed for the incoming EPS (event per second) and event size, and also evaluate the retention policy accordingly.
  1. On the Database cluster node1 server, create a folder for the database installer.

    2. For example:

    mkdir /opt/arcsight-db-tools
    /opt/arcsight-db-tools should not be under /root or /opt/vertica.
  2. From the master node where you performed the Downloading Installation Packages steps, copy the following directory on the Database cluster node1 server:

    3. {unzipped-installer-dir}/installers/database/db-installer_x.x.x-x.tar.gz file to the /opt/arcsight-db-tools

  3. To extract the installer file and place it in the correct directory, run the following commands:
    4. cd /opt/arcsight-db-tools
tar xvfz db-installer_x.x.x.x.tar.gz
  4. Edit the config/db_user.properties file. The hosts property is required.

    5. Property

    Description

    hosts

    A comma separated list of the database servers in IPv4 format (for example, 1.1.1.1,1.1.1.2,1.1.1.3).

    If it is necessary to construct the cluster, avoid using local loopback (localhost, 127.0.0.1, etc.).
  5. Install the database.
    6. ./db_installer install
  6. When prompted, create the database administrator user.

    7. The database administrator user account is used during database deployment, configuration, upgrade, and debugging. For security reasons, the platform deployed capabilities will not ask you for the credentials for this user.

    For a list of options that you can specify when installing the database, see Understanding the Database Installer Options.
  7. Create the schema.
    8. ./db_installer create-schema
  8. When prompted, create the following users:

    • App admin user: A regular database user granted elevated permissions for performing operations on the database to manage the database, schema, and resource pools. The credentials for this user will need to be provided later in the CDF Management Portal when you are deploying capabilities.

    • Search user: A regular database user with permissions restricted to event search operations. The credentials for this user will need to be provided later in the CDF Management Portal when you are deploying capabilities.

  9. Monitor your database cluster status constantly. For more information, see Monitoring the Database.
    • Database nodes status: Ensures all nodes are up
    • Database nodes storage status: Ensures storage is sufficient

 

Installing the Database in Cost Effective Long Term Retention Mode (Eon Mode)