🌞

19-11-27 W: similarity suggestions

GOAL: suggest similar from note as I'm writing it.

How close am I? Start with this doc:

.

ARCHITECTURE:

Link → representation (ping BERT as a service)

Get ml-box working!
Fix bugs, get link running.
Get whole text of file.
Find BERT as a service code, POST to it.

Embed all my Notion documents

DB table with their URL + Django admin page
Add the pages I want scraped
Add "scrape" action
Reproduce that error - seems like a lot of text is getting made?
Scrape document: get page, decide if DB or page. Print decision.
get or create Notion docs by URL... and/or ID?
get or create Text blobs by... text?
Users care about root nodes
Actually scrape everything!
Bookmarked docs: filterable, scrapeable, awesome!
Fix: web.models.NotionDocument.DoesNotExist: NotionDocument matching query does not exist.
Filter all should filter to anything
Fix: there only 10 results for People because it's choosing first view.
UPSERT text records
Make sure BERT works remotely!
Reinstall python, pip, django-numpy
Can't migrate with Numpy ArrayField. Fuck it - convert to list!
Create embeddings for all text as they're being scraped/processed!

BOOM!!! Amazing progress. Tomorrow: make a script that returns similarity for a document. Maybe a Bitbar plugin too?

Representation → similar (vector similarity).

BertClient connects over localhost:5555, which is forwarded to the server. Connection secured.

Bert server then tries to communicate back over 5556. To where? Localhost? A return IP? I have 5556 remotely forwarding back to local 5556.

would BertClient see it?

OK - BERT server receives a connection.

Trying to debug with https://jvns.ca/networking-zine.pdf sudo tcpdump port 5556 -w thing.pcap, but getting no packets. What am I misunderstanding? Read tcpdump, perhaps, and how it interacts with SSH forwarding.

OK - so I actually am receiving a bunch of traffic on 5556, but wait - is remotely forwarding to 5556 blocking my port here? No, I don't think so. 5555 has a listener on it, but not 5556.

Wait, I think it's because BertClient isn't listening on 5556? What is it supposed to be doing? Go read the code, and figure out what out_port does?

Next: I'm a bit stumped. Think about what could be going wrong. Seems like server is sending things back, but it's not getting to the client.

  • RemoteForward 5556.
    • Traceback (most recent call last):
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/bert_serving/client/__init__.py", line 211, in arg_wrapper
    return func(self, *args, **kwargs)
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/bert_serving/client/__init__.py", line 237, in server_config
    return jsonapi.loads(self._recv(req_id).content[1])
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/bert_serving/client/__init__.py", line 169, in _recv
    raise e
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/bert_serving/client/__init__.py", line 157, in _recv
    response = self.receiver.recv_multipart()
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/zmq/sugar/socket.py", line 475, in recv_multipart
    parts = [self.recv(flags, copy=copy, track=track)]
  File "zmq/backend/cython/socket.pyx", line 791, in zmq.backend.cython.socket.Socket.recv
  File "zmq/backend/cython/socket.pyx", line 827, in zmq.backend.cython.socket.Socket.recv
  File "zmq/backend/cython/socket.pyx", line 191, in zmq.backend.cython.socket._recv_copy
  File "zmq/backend/cython/socket.pyx", line 186, in zmq.backend.cython.socket._recv_copy
  File "zmq/backend/cython/checkrc.pxd", line 19, in zmq.backend.cython.checkrc._check_rc
zmq.error.Again: Resource temporarily unavailable
  • Neither.
    • Traceback (most recent call last):
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/bert_serving/client/__init__.py", line 211, in arg_wrapper
    return func(self, *args, **kwargs)
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/bert_serving/client/__init__.py", line 237, in server_config
    return jsonapi.loads(self._recv(req_id).content[1])
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/bert_serving/client/__init__.py", line 169, in _recv
    raise e
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/bert_serving/client/__init__.py", line 157, in _recv
    response = self.receiver.recv_multipart()
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/zmq/sugar/socket.py", line 475, in recv_multipart
    parts = [self.recv(flags, copy=copy, track=track)]
  File "zmq/backend/cython/socket.pyx", line 791, in zmq.backend.cython.socket.Socket.recv
  File "zmq/backend/cython/socket.pyx", line 827, in zmq.backend.cython.socket.Socket.recv
  File "zmq/backend/cython/socket.pyx", line 191, in zmq.backend.cython.socket._recv_copy
  File "zmq/backend/cython/socket.pyx", line 186, in zmq.backend.cython.socket._recv_copy
  File "zmq/backend/cython/checkrc.pxd", line 19, in zmq.backend.cython.checkrc._check_rc
zmq.error.Again: Resource temporarily unavailable
  • LocalForward 5556.

    • YUP! Makes sense, because both connections were initiated on the client. Awesome.

Well - I got something...!

WHILE WRITING A DOC - SEE WHAT'S SIMILAR!

Holy COW I can't believe how much better cosine similarity is than a simple dot product (for NLP - image interpretability research uses dot products). Cosine similarity normalizes their lengths, right?

These suggestions are just so awesome.