Tornado повышает StreamClosedError: поток закрыт

Question

Я использую торнадо и asyncmc для создания службы асинхронного API. Он работает отлично в течение нескольких недель, но сегодня проблема пришла. Почти все мои сервисы, кэшированные asyncmc, повышают StreamClosedError.

Из журнала торнадо, он сказал:

ile "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/web.py", line 1348, in _execute 
    result = yield result 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 807, in run 
    value = future.result() 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 209, in result 
    raise_exc_info(self._exc_info) 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 810, in run 
    yielded = self.gen.throw(*sys.exc_info()) 
    File "/opt/app/python/tv-appstore/handler/base.py", line 100, in wrapper 
    result = yield mc.get(self.cache_key) 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 807, in run 
    value = future.result() 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 209, in result 
    raise_exc_info(self._exc_info) 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 810, in run 
    yielded = self.gen.throw(*sys.exc_info()) 
    File "build/bdist.linux-x86_64/egg/asyncmc/client.py", line 63, in wrapper 
    res = yield func(self, conn, *args, **kwargs) 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 807, in run 
    value = future.result() 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 209, in result 
    raise_exc_info(self._exc_info) 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 810, in run 
    yielded = self.gen.throw(*sys.exc_info()) 
    File "build/bdist.linux-x86_64/egg/asyncmc/client.py", line 238, in get 
    result = yield self._multi_get(conn, self._key_type(key=key)) 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 807, in run 
    value = future.result() 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 209, in result 
    raise_exc_info(self._exc_info) 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 810, in run 
    yielded = self.gen.throw(*sys.exc_info()) 
    File "build/bdist.linux-x86_64/egg/asyncmc/client.py", line 278, in _multi_get 
    servers_resp = yield conn.send_cmd_all(cmd, stream=True) 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 807, in run 
    value = future.result() 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 209, in result 
    raise_exc_info(self._exc_info) 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 810, in run 
    yielded = self.gen.throw(*sys.exc_info()) 
    File "build/bdist.linux-x86_64/egg/asyncmc/pool.py", line 91, in send_cmd_all 
    server_resp = yield host.send_cmd(cmd, *arg, **kw) 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 807, in run 
    value = future.result() 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 209, in result 
    raise_exc_info(self._exc_info) 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 212, in wrapper 
    yielded = next(result) 
    File "build/bdist.linux-x86_64/egg/asyncmc/host.py", line 77, in send_cmd 
    yield self.stream.write(cmd) 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/iostream.py", line 356, in write 
    self._check_closed() 
    File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/iostream.py", line 864, in _check_closed 
    raise StreamClosedError("Stream is closed") 

Answer 1

Я имел такой же вопрос. Это может произойти, если вы установите minsize (для asyncmc.Client) более 1 и когда некоторые соединения в ConnectionPool были разбиты по причинам, не зависящим от используемой библиотеки (торнадо, asyncmc). Для решения этой проблемы я получаю значение из кэша по собственному методу:

self.mc_get(key) 

вместо

self.mc.get(key) 

и вызвать self.mc.pool.clear() для очистки пул соединений и начальное «полное повторное подключение».

Python код:

import tornado 
from tornado.iostream import StreamClosedError 

class SomeBaseHandler(tornado.web.RequestHandler): 

    def mc_get(self, key, default=None): 
     """ Additional interlayer for debug and logging requested keys from memcached 

     Args: 
      key (str): name of key in memcached 
      default: default value, returns if key not exists 

     Returns: 
      obj 
     """ 
     try: 
      return yield self.mc.get(key, default) 
     except StreamClosedError as e: 
      self.mc.pool.clear() 
      # Here you can raise own exception 
      return default 

    # … 

Я не использую рекурсивный вызов mc_get, чтобы избежать бесконечного цикла, без добавления дополнительного кода в метод.