Repository

Repository data source: rp

Repository data are raw events collected from a source control (repository) system.

Examples:

A user fetched files from a directory /project_files/linux/tools/
A user added files to a directory /depot/project5/java_source/

Information in this section pertain to the following repository systems and their versions:

Repository System	Version
GitHub Enterprise	2.21.0
Bitbucket Server	7.5.0
Perforce	2020.1

The repository systems store audit information in log files. The ArcSight FlexConnectors are installed and configured on the repository systems where they read the log files, filter the messages, tokenise them, then populate them in the default_secops_adm.Events table. For each of the repository systems and the specified versions, there is a corresponding configuration file (also referred to as a parser). The configuration file is a text file containing properties (name, value pairs) that describe how the FlexConnector parses event data.

The FlexConnector type that is used to process and parse the repository log files is the ArcSight FlexConnector Regex File.

Configuration Files

The configuration files provided in this section are designed only for the specified versions of the repository systems.

Configuration File for GitHub Enterprise 2.21.0

The configuration file that is used for GitHub Enterprise 2.21.0 is git.sdkrfilereader.properties.

text.qualifier="

comments.start.with=\#

trim.tokens=true

contains.empty.tokens=true



line.include.regex=(.+)"committer_date":"([^ ]+)(.+)"hostname":"([^,]+)"(.+)"program":("upload-pack"|"run-hook-postreceive")(.+)"

real_ip":"([^,]+)"(.+)"repo_name":"([^,]+)"(.+)"user_login":"([^,]+)"(.+)

regex=(.+)"committer_date":"([^ ]+)(.+)"hostname":"([^,]+)"(.+)"program":"([^,]+)"(.+)"real_ip":"([^,]+)"(.+)"repo_

name":"([^,]+)"(.+)"user_login":"([^,]+)"(.+)

token.count=13



token[0].name=CONSTANT1

token[0].type=String

token[1].name=EVENTTIME

token[1].type=Long

token[2].name=CONSTANT2

token[2].type=String

token[3].name=HOSTNAME

token[3].type=String

token[4].name=CONSTANT2

token[4].type=String

token[5].name=PROGRAM

token[5].type=String

token[6].name=CONSTANT3

token[6].type=String

token[7].name=REALIP

token[7].type=String

token[8].name=CONSTANT4

token[8].type=String

token[9].name=REPONAME

token[9].type=String

token[10].name=CONSTANT5

token[10].type=String

token[11].name=USERNAME

token[11].type=String

token[12].name=CONSTANT6

token[12].type=String



event.deviceVendor=__getVendor("GitHub")

event.deviceProduct=__stringConstant("GitGub Enterprise")

event.deviceVersion=__stringConstant("2.21.0")



event.deviceReceiptTime=__createLocalTimeStampFromSecondsSinceEpoch(EVENTTIME)

event.destinationUserName=USERNAME

event.deviceCustomString1=__toLowerCase(REPONAME)

event.deviceCustomString1Label=__stringConstant("RepositoryName")

event.deviceAction=__ifThenElse(PROGRAM,"run-hook-post-receive","receive-pack","upload-pack")

event.sourceAddress=__oneOfAddress(REALIP)

event.destinationHostName=__oneOfHostName(HOSTNAME)

event.name=__ifThenElse(PROGRAM,"run-hook-post-receive","receive-pack","upload-pack")

event.bytesOut=__safeToInteger(__regexToken(CONSTANT5,".+uploaded_bytes.:([^,]+)"))

#event.requestMethod=

#event.protocol=

#event.request=



event.categoryObject=__stringConstant("/Host/Resource")

event.categoryBehavior=__stringConstant("/Access")

event.categoryOutcome=__stringConstant("/Attempt")

event.categorySignificance=__stringConstant("/Informational")

event.categoryDeviceGroup=__stringConstant("Application")

event.categoryDeviceType=__stringConstant("Repository")

Configuration File for Bitbucket Server 7.5.0

The configuration file that is used for Bitbucket Server 7.5.0 is bitbucket.sdkrfilereader.properties.

text.qualifier="

comments.start.with=\#

trim.tokens=true

contains.empty.tokens=true



line.include.regex=(.+)\\|(.+)\\|(.+)\\|([^-]+)\\|(.+)\\|(.+git-upload-pack.+|.+git-receive-pack.+)\\|(.+)\\|(.+)\\|

(.+)\\|(.+)\\|(.+)\\|(.+)\\|(.+)\\|(.*)

regex=(.+)\\|(.+)\\|(.+)\\|(.*)\\|(.+)\\|(.*)\\|(.+)\\|(.+)\\|(.+)\\|(.+)\\|(.+)\\|(.+)\\|(.+)\\|(.*)



token.count=14



token[0].name=REALIP

token[0].type=String

token[1].name=PROTOCOL

token[1].type=String

token[2].name=REQUESTID

token[2].type=String

token[3].name=USERNAME

token[3].type=String

token[4].name=EVENTTIME

token[4].type=String

token[5].name=ACTION

token[5].type=String

token[6].name=REQUESTINFO

token[6].type=String

token[7].name=STATUS

token[7].type=String

token[8].name=BYTESREAD

token[8].type=String

token[9].name=BYTESWROTE

token[9].type=String

token[10].name=EXTRAINFO1

token[10].type=String

token[11].name=EXTRAINF02

token[11].type=String

token[12].name=EXTRAINF03

token[12].type=String

token[13].name=EXTRAINF04

token[13].type=String



event.deviceVendor=__getVendor("BitBucket")

event.deviceProduct=__stringConstant("BitBuket Server")

event.deviceVersion=__stringConstant("7.5.0")



event.deviceReceiptTime=__createOptionalTimeStampFromString(EVENTTIME,"yyyy-MM-dd HH:mm:ss,sss")

event.destinationUserName=USERNAME

event.deviceCustomString1=__toLowerCase(__regexToken(__regexToken(__split(ACTION," ",2),"(.*)\\.git(.+)"),".*\/(.+)"))

event.deviceCustomString2=__regexToken(__split(ACTION," ",2),"(\/.+)(\/git-upload-pack|\/git-receive-pack)")

event.deviceCustomString2Label=__stringConstant("RepositoryName")

event.name=__regexToken(__split(ACTION," ",2),".+\/(.+)")

event.sourceAddress=__oneOfAddress(REALIP)

event.sourceHostName=__oneOfHostName(REALIP)

event.deviceAction=__regexToken(__split(ACTION," ",2),".+\/(.+)")

event.bytesIn=__safeToInteger(BYTESREAD)

event.bytesOut=__safeToInteger(BYTESWROTE)

event.requestMethod=__ifThenElse(__contains(ACTION,"POST"),"true","POST","GET")

event.requestUrl=__split(ACTION," ",2)



event.categoryObject=__stringConstant("/Host/Resource")

event.categoryBehavior=__stringConstant("/Access")

event.categoryOutcome=__ifThenElse(STATUS,"200","/Success",__ifThenElse(STATUS,"401","/Denied","/Attempt"))

event.categorySignificance=__stringConstant("/Informational")

event.categoryDeviceGroup=__stringConstant("Application")

event.categoryDeviceType=__stringConstant("Repository")

Configuration File for Perforce 2020.1

The configuration file that is used for Perforce 2020.1 is perforce.sdkrfilereader.properties.

text.qualifier="
comments.start.with=\#
trim.tokens=true
contains.empty.tokens=true

regex=(.+)\\s(.+)\\s(.+)\\s(.+)\\s(.+)\\s(.+)

token.count=6

token[0].name=EVENTDATE
token[0].type=String
token[1].name=EVENTTIME
token[1].type=String
token[2].name=USER
token[2].type=String
token[3].name=CLIENTIP
token[3].type=String
token[4].name=ACTION
token[4].type=String
token[5].name=RESOURCE
token[5].type=String

event.deviceVendor=__getVendor("Perforce")
event.deviceProduct=__stringConstant("Perforce")
event.deviceVersion=__stringConstant("2020.1")

event.deviceReceiptTime=__createOptionalTimeStampFromString(__concatenate(EVENTDATE,EVENTTIME),"yyyy/MM/ddHH:mm:ss")
event.destinationUserName=USER

#########################################################################
#1.\/\/([^\/]+)\/([^\/]+)\/([^\/]+).*","/","//","")
# will return max of depth 4
# __regexTokenFindAndJoin(RESOURCE,"\/\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?","/","//","")
# eg //csvr/A/B/C
# //csrv/main/null
# //csrv/null/null
# //csrv/A/master
#2.\/\/(.*)(?=\/main$|\/null$|\/rel$|\/master$)
#__regexToken(__regexTokenFindAndJoin(RESOURCE,"\/\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?","/","//",""),"\/\/(.*)
(?=\/main$|\/null$|\/rel$|\/master$)")
#eg.returns all info nothign with main/null/rel/master
#3. remove version if any
#__regexToken(__ifGreaterOrEqual(__length(__regexToken(__regexTokenFindAndJoin(RESOURCE,"\/\/([^\/]+)?\/?([^\/]+)?\/?
([^\/]+)?","/","//",""),"\/\/(.*)(?=\/main$|\/null$|\/rel$|\/master$)")),"1",__regexToken(__regexTokenFindAndJoin
(RESOURCE,"\/\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?","/","//",""),"\/\/(.*)(?=\/main$|\/null$|\/rel$|\/master$)"),__
regexTokenFindAndJoin(RESOURCE,"\/\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?","/","//","")),"(.*)[#\/][\\d.]+")
#eg.//crsv/A/12.3
# //crsv/A#1.2
##########################################################################

event.deviceCustomString1=__ifGreaterOrEqual(__length(__regexToken(__ifGreaterOrEqual(__length(__regexToken(__
regexTokenFindAndJoin(RESOURCE,"\/\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?","/","//",""),"(\/\/.*)
(?=\/main$|\/null$|\/rel$|\/master$)")),"1",__regexToken(__regexTokenFindAndJoin(RESOURCE,"\/\/([^\/]+)?\/?([^\/]+)?\/?
([^\/]+)?","/","//",""),"(\/\/.*)(?=\/main$|\/null$|\/rel$|\/master$)"),__regexTokenFindAndJoin(RESOURCE,"\/\/
([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?","/","//","")),"(.*)[#\/][\\d.]+")),"1",__regexToken(__ifGreaterOrEqual(__length(__
regexToken(__regexTokenFindAndJoin(RESOURCE,"\/\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?","/","//",""),"(\/\/.*)
(?=\/main$|\/null$|\/rel$|\/master$)")),"1",__regexToken(__regexTokenFindAndJoin(RESOURCE,"\/\/([^\/]+)?\/?([^\/]+)?\/?
([^\/]+)?","/","//",""),"(\/\/.*)(?=\/main$|\/null$|\/rel$|\/master$)"),__regexTokenFindAndJoin(RESOURCE,"\/\/
([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?","/","//","")),"(.*)[#\/][\\d.]+"),__ifGreaterOrEqual(__length(__regexToken(__
regexTokenFindAndJoin(RESOURCE,"\/\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?","/","//",""),"(\/\/.*)
(?=\/main$|\/null$|\/rel$|\/master$)")),"1",__regexToken(__regexTokenFindAndJoin(RESOURCE,"\/\/([^\/]+)?\/?([^\/]+)?\/?
([^\/]+)?","/","//",""),"(\/\/.*)(?=\/main$|\/null$|\/rel$|\/master$)"),__regexTokenFindAndJoin(RESOURCE,"\/\/
([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?","/","//","")))
event.deviceCustomString2=RESOURCE
event.deviceAction=ACTION
event.sourceAddress=__oneOfAddress(CLIENTIP)
event.sourceHostName=__oneOfHostName(CLIENTIP)
event.name=ACTION

event.categoryObject=__stringConstant("Host/Resource")
event.categoryBehavior=__stringConstant("Access")
event.categoryOutcome=__stringConstant("/Attempt")
event.categorySignificance=__stringConstant("/Informational")
event.categoryDeviceGroup=__stringConstant("Application")
event.categoryDeviceType=__stringConstant("Repository")

You can also create or customize the configuration files for other versions of the repository systems. For more information, see ArcSight FlexConnector Developer’s Guide.

FlexConnector Installation and Configuration

To install and configure a FlexConnector, see ArcSight FlexConnector Developer’s Guide.

Ensure the following when you install and configure the FlexConnector:

Select ArcSight FlexConnector Regex File as the Connector Type.
When adding the parameters information, specify the following:
- Select Log Unparsed Events as False.
- Provide the absolute path and the repository log file name that the FlexConnector needs to read in the Log File Name field.
  
  For example:
  c:\temp\sample_data.log
- For the Configuration File field, depending on the repository on which you are installing the FlexConnector, specify only git, bitbucket, or perforce.
  
  For example, for the GitHub Enterprise repository, you must specify only git. The suffix .sdkrfilereader.properties is appended automatically. The configuration file name now is git.sdkrfilereader.properties.
When configuring the destination, select either CEF File or Transformation Hub as the destination. For more information, see SmartConnector Installation and User Guide.

Post-Installation Tasks

After you install and configure the FlexConnector and before you run the FlexConnector, copy the desired configuration (parser) files in the ARCSIGHT_HOME\user\agent\ flexagent location.

Repository Schema

The following table describes the default_secops_adm.Events table columns for Repository data.

Column Name	Type	Required (Y/N)	Description	Example
deviceAction	Varchar	Y	The action performed on the device.	upload-pack
deviceCustomString1	Varchar	Y	The device involved in the event. Typically a file path. Can be any string identifying a repository. Secondary entity for the rp data source	dev3/rel/hydra
deviceReceiptTime	Integer	Y	The time at which the event related to the activity was received.	1592839336200 Equivalent GMT -2020-06-22 15:22:00
destinationUserName	Varchar	Y	The user involved in the event. Primary entity.	john.legget
deviceVendor	Varchar	Y	The device vendor of the client.	GitHub
deviceProduct	Varchar	N	The device product of the client.	GitHub Server
deviceVersion	Integer	N	The device version.	2.21.0
categoryObject	Varchar	N	The type of the object.	Host/Resource
categoryBehavior	Varchar	N	The action or behavior associated with the event.	/Access
categoryOutcome	Varchar	Y	The outcome of the event.	/Attempt
cateorySignificance	Varchar	N	The significance of the event.	/Informational
categoryDeviceGroup	Varchar	Y	The type of events for the device.	Application
categoryDeviceType	Varchar	N	The events generated by the device type irrespective of the device group the events belong to.	Repository
sourceAddressBin	Varchar	N	The IP address of the user involved in the event.	78.1.198.82
bytesOut/bytesIn	Integer	N	The size of data (in bytes) related to the action performed on the project.	2203