Setup

Configure Vector
  1. Install Vector

    If you haven’t already, install Vector:

    curl https://sh.vector.dev -sSf | sh
    

    For alternate installation instructions, see Installation from Vector.

  2. Configure Vector with Logz.io sink

    Add this code block to your Vector configuration file. We recommend the configuaration shown in the code block.

    Find the complete configuration docs at http sink from Vector.

    [sinks.logzio]
      # REQUIRED - General
      type = "http" # Don't change this setting
      inputs = ["YOUR_SOURCE_ID"]
      encoding = "ndjson" # enum: "ndjson" or "text"
    
      # More information on uri below this code block
      uri = "https://<<LISTENER-HOST>>?token=<<SHIPPING-TOKEN>>&type=vector"
    
      # OPTIONAL - General
      compression = "gzip" # no default, must be: "gzip" (if supplied)
    
      # OPTIONAL - Batching
      batch_size = 9000000 # bytes - Logz.io max batch is 10MB
      batch_timeout = 3
    
      # OPTIONAL - Buffer
      [sinks.logzio.buffer]
        type = "disk" # default, enum: "memory" or "disk"
        when_full = "block" # default, enum: "block" or "drop_newest"
        max_size = 104900000 # no default, bytes(104.9mb), relevant when type = "disk"
    

    Parameters

    uri
    Your Logz.io region’s listener URL account token, and log type.
    Replace <<LISTENER-HOST>> with your region’s listener host (for example, listener.logz.io). For more information on finding your account’s region, see Account region.
    Replace <<SHIPPING-TOKEN>> with the token of the account you want to ship to.
  3. Run Vector

    vector --config path/to/your/vector.toml
    
  4. Check Logz.io for your logs

    Give your logs some time to get from your system to ours, and then open Kibana.

    If you still don’t see your logs, see log shipping troubleshooting.

Try it out

  1. Install Vector
  2. Create a vector.toml configuration file:

    touch vector.toml
    
  3. Create a folder for Vector logs:

      mkdir vector
    
  4. Configure vector.toml with the configuration below and replace your parameters:

    # Set global options
    data_dir = "./vector"
    
    [sources.in]
      type = "stdin"
    
    # Structure and parse the data
    [transforms.apache_parser]
        inputs = ["in"]
        type   = "regex_parser"
        regex    = '^(?P<host>[\w\.]+) - (?P<user>[\w]+) (?P<bytes_in>[\d]+) \[(?P<timestamp>.*)\] "(?P<method>[\w]+) (?P<path>.*)" (?P<status>[\d]+) (?P<bytes_out>[\d]+)$'
    
    # Set Logz.io sink
    [sinks.logzio]
      # REQUIRED - General
      type = "http" # must be: "http"
      inputs = ["YOUR_SOURCE_ID"]
      encoding = "ndjson" # enum: "ndjson" or "text"
      uri = "https://<YOUR_LOGZIO_HOST>?token=<YOUR_LOGZIO_SHIPPING_TOKEN>&type=vector"
    
      # OPTIONAL - General
      compression = "gzip" # no default, must be: "gzip" (if supplied)
    
      # OPTIONAL - Batching
      batch_size = 9000000 # bytes - Logz.io max batch is 10MB
      batch_timeout = 3
    
      # OPTIONAL - Buffer
      [sinks.logzio.buffer]
        type = "disk" # default, enum: "memory" or "disk"
        when_full = "block" # default, enum: "block" or "drop_newest"
        max_size = 104900000 # no default, bytes(104.9mb), relevant when type = "disk"
    
  5. Run Vector:

    echo '172.128.80.109 - Bins5273 656 [2019-05-03T13:11:48-04:00] "PUT /mesh" 406 10272' \
      | vector --config ./vector.toml
    
  6. Check Logz.io for your logs