Hi,
any idea for the error bellow?
WARNING:root:Setting up a new session...
Exception in user code:
------------------------------------------------------------
Traceback (most recent call last):
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection
raise err
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/connectionpool.py", line 603, in urlopen
chunked=chunked)
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/connectionpool.py", line 355, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.6/http/client.py", line 1254, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1300, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1249, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/lib/python3.6/http/client.py", line 974, in send
self.connect()
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/connection.py", line 183, in connect
conn = self._new_conn()
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f6ed6b2a5f8>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/../lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/connectionpool.py", line 641, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /env/scv2_aux_ft_gu (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ed6b2a5f8>: Failed to establish a new connection: [Errno 111] Connection refused',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/visdom/init.py", line 548, in _send
data=json.dumps(msg),
File "/home/ant/env/.../lib/python3.6/site-packages/requests/sessions.py", line 581, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/ant/env/.../lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/home/ant/env/..../lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/home/ant/env/.../lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /env/scv2_aux_ft_gu (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ed6b2a5f8>: Failed to establish a new connection: [Errno 111] Connection refused',))
WARNING:visdom:Without the incoming socket you cannot receive events from the server or register event handlers to your Visdom client.
Exception in user code:
------------------------------------------------------------
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection
raise err
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 603, in urlopen
chunked=chunked)
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 355, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.6/http/client.py", line 1254, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1300, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1249, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/lib/python3.6/http/client.py", line 974, in send
self.connect()
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connection.py", line 183, in connect
conn = self._new_conn()
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f6ed6034208>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 641, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /close (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ed6034208>: Failed to establish a new connection: [Errno 111] Connection refused',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/visdom/init.py", line 548, in _send
data=json.dumps(msg),
File "/home/ant/env/.../lib/python3.6/site-packages/requests/sessions.py", line 581, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/ant/env/.../lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/home/ant/env/.../lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/home/ant/env/.../lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /close (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ed6034208>: Failed to establish a new connection: [Errno 111] Connection refused',))
Exception in user code:
------------------------------------------------------------
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection
raise err
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 603, in urlopen
chunked=chunked)
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 355, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.6/http/client.py", line 1254, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1300, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1249, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/lib/python3.6/http/client.py", line 974, in send
self.connect()
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connection.py", line 183, in connect
conn = self._new_conn()
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f6ed60344e0>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 641, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /events (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ed60344e0>: Failed to establish a new connection: [Errno 111] Connection refused',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/visdom/init.py", line 548, in _send
data=json.dumps(msg),
File "/home/ant/env/.../lib/python3.6/site-packages/requests/sessions.py", line 581, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/ant/env/.../lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/home/ant/env/.../lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/home/ant/env/.../lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /events (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ed60344e0>: Failed to establish a new connection: [Errno 111] Connection refused',))
Transferring Encoder weights ...
Classifier(
(lockdrop): LockedDropout()
(idrop): Dropout(p=0.1)
(hdrop): Dropout(p=0.1)
(drop): Dropout(p=0.1)
(embed): Embed(
(embedding): Embedding(70004, 400)
(dropout): Dropout(p=0.3)
(noise): GaussianNoise (mean=0.0, stddev=0.1)
)
(bottom_rnn): RNNModule(
(rnn): LSTM(400, 1000, num_layers=2, batch_first=True, dropout=0.5)
)
(top_rnn): RNNModule(
(rnn): LSTM(1000, 100, batch_first=True, dropout=0.3)
)
(attention): SelfAttention(
(attention): Sequential(
(0): Linear(in_features=100, out_features=1, bias=True)
(1): Tanh()
(2): Dropout(p=0.3)
)
(softmax): Softmax()
)
(vocab): Linear(in_features=1000, out_features=70004, bias=True)
(classes): Linear(in_features=100, out_features=2, bias=True)
)
THCudaCheck FAIL file=/pytorch/aten/src/THC/generic/THCStorage.cu line=58 error=2 : out of memory
Traceback (most recent call last):
File "models/run_clf.py", line 37, in
opts=opts, transfer=transfer)
File "/home/ant/multitask/siatl/models/../models/sent_clf.py", line 155, in sent_clf
train_loss = trainer.train_epoch()
File "/home/ant/multitask/siatl/models/../models/sent_clf_trainer.py", line 85, in train_epoch
loss_sum.backward()
File "/home/ant/env/.../lib/python3.6/site-packages/torch/tensor.py", line 93, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/ant/env/.../lib/python3.6/site-packages/torch/autograd/init.py", line 89, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/generic/THCStorage.cu:58