Skip to content

Update azure_kusto.md for buffering commits #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 14, 2025
Merged

Update azure_kusto.md for buffering commits #1

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions pipeline/outputs/azure_kusto.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,19 @@ By default, Kusto will insert incoming ingestions into a table by inferring the
| include_time_key | If enabled, a timestamp is appended to output. The key name is used `time_key` property. | `On` |
| time_key | The key name of time. If `include_time_key` is false, This property is ignored. | `timestamp` |
| workers | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |
| buffering_enabled | _Optional_ - Enable buffering into disk before ingesting into Azure Kusto. | `Off` |
| buffer_dir | _Optional_ - When buffering is turned ON, specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-kusto/` |
| upload_timeout | _Optional_ - When buffering is turned ON, specifies a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than x minutes and haven't reached upload_file_size limit yet. | `30m` |
| upload_file_size | _Optional_ - When buffering is turned ON, specifies the size of files to be uploaded in MBs. | `200MB` |
| azure_kusto_buffer_key | _Optional_ - When buffering is turned ON, set the azure kusto buffer key which needs to be specified when using multiple instances of azure kusto output plugin and buffering is enabled. | `key` |
| store_dir_limit_size | _Optional_ - When buffering is turned ON, set the max size of the buffer directory. | `8GB` |
| buffer_file_delete_early | _Optional_ - When buffering is turned ON, whether to delete the buffered file early after successful blob creation. | `Off` |
| unify_tag | _Optional_ - This creates a single buffer file when the buffering mode is ON. | `On` |
| blob_uri_length | _Optional_ - Set the length of generated blob uri before ingesting to kusto. | `64` |
| scheduler_max_retries | _Optional_ - When buffering is turned ON, Set the maximum number of retries for ingestion using the scheduler. | `3` |
| use_imds | _Optional_ - Whether to use IMDS to retrieve oauth token. | `Off` |
| delete_on_max_upload_error | _Optional_ - When buffering is turned ON, Whether to delete the buffer file on maximum upload errors. | `Off` |
| io_timeout | _Optional_ - Configure the HTTP IO timeout for uploads. | `60s` |

### Configuration File

Expand All @@ -80,6 +93,19 @@ Get started quickly with this configuration file:
Database_Name <database_name>
Table_Name <table_name>
Ingestion_Mapping_Reference <mapping_name>
buffering_enabled On
upload_timeout 2m
upload_file_size 125M
azure_kusto_buffer_key kusto1
buffer_file_delete_early Off
unify_tag On
use_imds Off
buffer_dir /var/log/
store_dir_limit_size 16GB
blob_uri_length 128
scheduler_max_retries 3
delete_on_max_upload_error Off
io_timeout 60s
```

## Troubleshooting
Expand Down