Class: BigQueryEventStreamer

google/bigquery_utils~BigQueryEventStreamer

new BigQueryEventStreamer(configFile, chunkSize)

BigQuery implementation of EventStreamer.

Streams log lines into BigQuery tables using insertAll() API command, using Redis to store lines locally to reduce the number of calls required.

USAGE

This implementation requires external configuration file defining field schema for each "type" of event being logged, as BQ table expects fixed field schema with rigid typing. It is recommended that you write custom logging methods to enforce each event-type schema for desired events.

Ex:

Given a config file containing:

{"projectId": "project-id-781",
 "event_tables": {
  "SOME_EVENT": {
    "tableId": "test_events",
    "datasetId": "test_events_dataset",
    "schema": {
      "fields": [
        {"name": "level","type": "STRING"},
        {"name": "msg","type": "STRING"},
        {"name": "tstamp","type": "TIMESTAMP"},
        {"name": "val1","type": "STRING"},
        {"name": "val2","type": "STRING"}
      ]
    }
  }
}

You would log SOME_EVENT type events to the table "test_events" using the following call to the log method:

$ logger.log("info", msg, {type: "SOME_EVENT, val2: "a val", val2: "another val"})

IMPORTANT: If the meta key type is not found, or the given type is not configured in the provided config file, the event will be ignored in this transport.

Parameters:
Name Type Description
configFile String

path to BigQuery event schema config file. See specifications for this file above.
String

Path to JWT secrets JSON file.
chunkSize Number

number of rows to cache per event-type before sending to BigQuery. Default is 500 (maxRequestSize)
Source:

Methods

getEventSchema()

Hook to parse config file into eventSchema format accepted by transport class.

Source:

streamEvents(event_type, cached_rows, callback)

Executes API call to BigQuery.insertAll()

Generates a single HTTP POST request to Google BigQuery REST endpoint containing all rows in cache, writing to table & dataset specified in config per event_type

Parameters:
Name Type Description
event_type String

event type indicated by meta['type'] val passed to log method
cached_rows Array

array of row objects
callback function
Source: