Как указать самоподписанный SSL CERT для td-agent/fluentd?

Question

Я продолжал получать следующее сообщение об ошибке:

2015-06-16 05:21:17 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-06-16 05:21:17 +0000 error_class="Faraday::SSLError" error="Unable to verify certificate, please set `Excon.defaults[:ssl_ca_path] = path_to_certs`, `ENV['SSL_CERT_DIR'] = path_to_certs`, `Excon.defaults[:ssl_ca_file] = path_to_file`, `ENV['SSL_CERT_FILE'] = path_to_file`, `Excon.defaults[:ssl_verify_callback] = callback` (see OpenSSL::SSL::SSLContext#verify_callback), or `Excon.defaults[:ssl_verify_peer] = false` (less secure)." plugin_id="object:127283c" 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/ssl_socket.rb:129:in `connect' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/ssl_socket.rb:129:in `initialize' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/connection.rb:387:in `new' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/connection.rb:387:in `socket' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/connection.rb:106:in `request_call' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/middlewares/mock.rb:47:in `request_call' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/middlewares/instrumentor.rb:22:in `request_call' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/middlewares/base.rb:15:in `request_call' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/middlewares/base.rb:15:in `request_call' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/middlewares/base.rb:15:in `request_call' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/connection.rb:233:in `request' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.1/lib/faraday/adapter/excon.rb:54:in `call' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.1/lib/faraday/rack_builder.rb:139:in `build_response' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.1/lib/faraday/connection.rb:377:in `run_request' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/http/faraday.rb:21:in `block in perform_request' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:190:in `call' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:190:in `perform_request' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/client.rb:119:in `perform_request' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/actions/ping.rb:20:in `block in ping' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/utils.rb:189:in `__rescue_from_not_found' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/actions/ping.rb:19:in `ping' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-0.9.0/lib/fluent/plugin/out_elasticsearch.rb:64:in `client' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-0.9.0/lib/fluent/plugin/out_elasticsearch.rb:171:in `rescue in send' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-0.9.0/lib/fluent/plugin/out_elasticsearch.rb:169:in `send' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-0.9.0/lib/fluent/plugin/out_elasticsearch.rb:163:in `write' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/buffer.rb:325:in `write_chunk' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/buffer.rb:304:in `pop' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:321:in `try_flush' 
    2015-06-16 05:21:17 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:140:in `run' 
2015-06-16 05:21:18 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-06-16 05:21:19 +0000 error_class="Faraday::SSLError" error="Unable to verify certificate, please set `Excon.defaults[:ssl_ca_path] = path_to_certs`, `ENV['SSL_CERT_DIR'] = path_to_certs`, `Excon.defaults[:ssl_ca_file] = path_to_file`, `ENV['SSL_CERT_FILE'] = path_to_file`, `Excon.defaults[:ssl_verify_callback] = callback` (see OpenSSL::SSL::SSLContext#verify_callback), or `Excon.defaults[:ssl_verify_peer] = false` (less secure)." plugin_id="object:127283c" 
    2015-06-16 05:21:18 +0000 [warn]: suppressed same stacktrace 
2015-06-16 05:21:20 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-06-16 05:21:24 +0000 error_class="Faraday::SSLError" error="Unable to verify certificate, please set `Excon.defaults[:ssl_ca_path] = path_to_certs`, `ENV['SSL_CERT_DIR'] = path_to_certs`, `Excon.defaults[:ssl_ca_file] = path_to_file`, `ENV['SSL_CERT_FILE'] = path_to_file`, `Excon.defaults[:ssl_verify_callback] = callback` (see OpenSSL::SSL::SSLContext#verify_callback), or `Excon.defaults[:ssl_verify_peer] = false` (less secure)." plugin_id="object:127283c" 
    2015-06-16 05:21:20 +0000 [warn]: suppressed same stacktrace 
^C2015-06-16 05:21:22 +0000 [info]: shutting down fluentd 
2015-06-16 05:21:23 +0000 [warn]: before_shutdown failed error="Unable to verify certificate, please set `Excon.defaults[:ssl_ca_path] = path_to_certs`, `ENV['SSL_CERT_DIR'] = path_to_certs`, `Excon.defaults[:ssl_ca_file] = path_to_file`, `ENV['SSL_CERT_FILE'] = path_to_file`, `Excon.defaults[:ssl_verify_callback] = callback` (see OpenSSL::SSL::SSLContext#verify_callback), or `Excon.defaults[:ssl_verify_peer] = false` (less secure)." 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/ssl_socket.rb:129:in `connect' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/ssl_socket.rb:129:in `initialize' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/connection.rb:387:in `new' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/connection.rb:387:in `socket' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/connection.rb:106:in `request_call' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/middlewares/mock.rb:47:in `request_call' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/middlewares/instrumentor.rb:22:in `request_call' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/middlewares/base.rb:15:in `request_call' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/middlewares/base.rb:15:in `request_call' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/middlewares/base.rb:15:in `request_call' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/excon-0.45.3/lib/excon/connection.rb:233:in `request' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.1/lib/faraday/adapter/excon.rb:54:in `call' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.1/lib/faraday/rack_builder.rb:139:in `build_response' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.1/lib/faraday/connection.rb:377:in `run_request' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/http/faraday.rb:21:in `block in perform_request' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:190:in `call' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:190:in `perform_request' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/client.rb:119:in `perform_request' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/actions/ping.rb:20:in `block in ping' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/utils.rb:189:in `__rescue_from_not_found' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/actions/ping.rb:19:in `ping' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-0.9.0/lib/fluent/plugin/out_elasticsearch.rb:64:in `client' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-0.9.0/lib/fluent/plugin/out_elasticsearch.rb:171:in `rescue in send' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-0.9.0/lib/fluent/plugin/out_elasticsearch.rb:169:in `send' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-0.9.0/lib/fluent/plugin/out_elasticsearch.rb:163:in `write' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/buffer.rb:325:in `write_chunk' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/buffer.rb:304:in `pop' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_memory.rb:93:in `block in before_shutdown' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_memory.rb:89:in `before_shutdown' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:396:in `before_shutdown' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:160:in `block in run' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:159:in `synchronize' 
    2015-06-16 05:21:23 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:159:in `run' 
2015-06-16 05:21:23 +0000 [info]: process finished code=0 

Я попытался установить следующие переменные окружения:

• настройки SSL_CERT_FILE=/path/to/my/cert.crt
• настройки SSL_CERT_DIR=/path/to/my
• настройки SSL_CERT_DIR=/etc/ssl/certs
• установка SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

Я попытался создать /home/td-agent/.bash_profile и обновить файлы /home/td-agent/.bashrc для экспорта аналогичных переменных окружения. Когда я запускаю службу td-agent, я все равно получаю эту ошибку.

Я попытался запустить службу td-agent в командной строке с указанными ранее env vars.

У меня есть dpkg-reconfigure ca-certificates с моим самоподписанным сертификатом соответствующего места. /usr/share/ca-certificates/<my cert dir>/my-self-signed-cert.crt

Я попытался запустить td-agent без SSL_CERT_DIR или SSL_CERT_FILE env vars.

NB. Я смог закрутить https://my.elasticsearch.myltd и получить ответ 200, поэтому я знаю, что os (Ubuntu) взял мой сертификат.

Что касается версий, я бегу Ubuntu 14.04 LTS и когда я начинаю TD-агент, следующий выводится в лог-файл /var/log/td-agent/td-agent.log:

2015-06-16 05:07:05 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf" 
2015-06-16 05:07:05 +0000 [info]: starting fluentd-0.12.7 
2015-06-16 05:07:06 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0' 
2015-06-16 05:07:06 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6' 
2015-06-16 05:07:06 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '0.9.0' 
2015-06-16 05:07:06 +0000 [info]: gem 'fluent-plugin-mongo' version '0.7.8' 
2015-06-16 05:07:06 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1' 
2015-06-16 05:07:06 +0000 [info]: gem 'fluent-plugin-s3' version '0.5.9' 
2015-06-16 05:07:06 +0000 [info]: gem 'fluent-plugin-s3' version '0.5.7' 
2015-06-16 05:07:06 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14' 
2015-06-16 05:07:06 +0000 [info]: gem 'fluent-plugin-td' version '0.10.26' 
2015-06-16 05:07:06 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.0' 
2015-06-16 05:07:06 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1' 
2015-06-16 05:07:06 +0000 [info]: gem 'fluentd' version '0.12.7' 

Answer 1

нашел ответ здесь: https://groups.google.com/forum/#!topic/fluentd/z-1vIsQ4kHU

Выключается, чтобы установить переменные среды для службы td-agent, вам необходимо экспортировать переменные в файл /etc/default/td-agent.

Например:

# This file is sourced by /bin/sh from /etc/init.d/td-agent 
# Options to pass to td-agent 
DAEMON_ARGS="" 
export SSL_CERT_DIR=/etc/ssl/certs 
export MYOTHERVAR=jazzjazzjazz