Data Schema for the Spot Suspicious Connects Analyses

Three kinds of network logs can be analyzed by the Spot Suspicious Connects analyses: Netflow records, DNS queries, and proxy logs.

Data from these logs is expected to have been stored in parquet files on HDFS with column structure extending the following schema.

Netflow Data

• trhour: The hour of the day in which the flow was logged, Integer.
• sip: The source IP address of the flow, String.
• dip: The destination IP address of the flow, String.
• sport: The source port of the flow, Integer
• dport: The destination port of the flow, Integer
• proto: The protocol used by the flow, String
• ipkt: The flow's packet count, Long.
• ibyt: The flow's byte count, Long.

DNS Data

• frame_time: Frame timestamp of query, String.
• unix_tstamp: Unix timestamp of query, String.
• frame_len: Frame length, Integer.
• ip_dst: IP address of client making query, String.
• dns_qry_name: Name of DNS query, String.
• dns_qry_class: Class of DNS query, String.
• dns_qry_type: Type of DNS query, Integer.
• dns_qry_rcode: DNS query response code, Integer.

Proxy Data

• p_date: Date of the query, String.
• p_time: Time of the query, String. (FORMAT?)
• clientip: IP address of client making proxy request, String.
• host: Host of request, String.
• reqmethod: Request method, String.
• useragent: User agent, String.
• resconttype: Response content type, String.
• respcode: Response code, String.
• fulluri: Full URI of request, String.