19-11-27 W: similarity suggestions

GOAL: suggest similar from note as I'm writing it.

How close am I? Start with this doc: Research vs prototyping: research is deeper.

ARCHITECTURE:

Link → representation (ping BERT as a service)

~~Get ml-box working!~~

~~Fix bugs, get link running.~~

~~Get whole text of file.~~

~~Find BERT as a service code, POST to it.~~

Embed all my Notion documents

~~DB table with their URL + Django admin page~~

~~Add the pages I want scraped~~

~~Add "scrape" action~~

~~Reproduce that error - seems like a lot of text is getting made?~~

~~Scrape document: get page, decide if DB or page. Print decision.~~

~~get or create Notion docs by URL... and/or ID?~~

~~get or create Text blobs by... text?~~

~~Users care about root nodes~~

~~Actually scrape everything!~~

~~Bookmarked docs: filterable, scrapeable, awesome!~~

~~Fix: web.models.NotionDocument.DoesNotExist: NotionDocument matching query does not exist.~~

~~Filter all should filter to anything~~

~~Fix: there only 10 results for People because it's choosing first view.~~

~~UPSERT text records~~

~~Make sure BERT works remotely!~~

~~Reinstall python, pip, django-numpy~~

~~Can't migrate with Numpy ArrayField. Fuck it - convert to list!~~

~~Create embeddings for all text as they're being scraped/processed!~~

BOOM!!! Amazing progress. Tomorrow: make a script that returns similarity for a document. Maybe a Bitbar plugin too?

Representation → similar (vector similarity).

BertClient connects over localhost:5555, which is forwarded to the server. Connection secured.

Bert server then tries to communicate back over 5556. To where? Localhost? A return IP? I have 5556 remotely forwarding back to local 5556.

would BertClient see it?

OK - BERT server receives a connection.

Trying to debug with https://jvns.ca/networking-zine.pdf sudo tcpdump port 5556 -w thing.pcap, but getting no packets. What am I misunderstanding? Read tcpdump, perhaps, and how it interacts with SSH forwarding.

OK - so I actually am receiving a bunch of traffic on 5556, but wait - is remotely forwarding to 5556 blocking my port here? No, I don't think so. 5555 has a listener on it, but not 5556.

Wait, I think it's because BertClient isn't listening on 5556? What is it supposed to be doing? Go read the code, and figure out what out_port does?

Next: I'm a bit stumped. Think about what could be going wrong. Seems like server is sending things back, but it's not getting to the client.

RemoteForward 5556.

Traceback (most recent call last):
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/bert_serving/client/__init__.py", line 211, in arg_wrapper
    return func(self, *args, **kwargs)
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/bert_serving/client/__init__.py", line 237, in server_config
    return jsonapi.loads(self._recv(req_id).content[1])
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/bert_serving/client/__init__.py", line 169, in _recv
    raise e
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/bert_serving/client/__init__.py", line 157, in _recv
    response = self.receiver.recv_multipart()
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/zmq/sugar/socket.py", line 475, in recv_multipart
    parts = [self.recv(flags, copy=copy, track=track)]
  File "zmq/backend/cython/socket.pyx", line 791, in zmq.backend.cython.socket.Socket.recv
  File "zmq/backend/cython/socket.pyx", line 827, in zmq.backend.cython.socket.Socket.recv
  File "zmq/backend/cython/socket.pyx", line 191, in zmq.backend.cython.socket._recv_copy
  File "zmq/backend/cython/socket.pyx", line 186, in zmq.backend.cython.socket._recv_copy
  File "zmq/backend/cython/checkrc.pxd", line 19, in zmq.backend.cython.checkrc._check_rc
zmq.error.Again: Resource temporarily unavailable

Neither.

Traceback (most recent call last):
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/bert_serving/client/__init__.py", line 211, in arg_wrapper
    return func(self, *args, **kwargs)
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/bert_serving/client/__init__.py", line 237, in server_config
    return jsonapi.loads(self._recv(req_id).content[1])
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/bert_serving/client/__init__.py", line 169, in _recv
    raise e
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/bert_serving/client/__init__.py", line 157, in _recv
    response = self.receiver.recv_multipart()
  File "/Users/jasonbenn/.pyenv/versions/worldview/lib/python3.7/site-packages/zmq/sugar/socket.py", line 475, in recv_multipart
    parts = [self.recv(flags, copy=copy, track=track)]
  File "zmq/backend/cython/socket.pyx", line 791, in zmq.backend.cython.socket.Socket.recv
  File "zmq/backend/cython/socket.pyx", line 827, in zmq.backend.cython.socket.Socket.recv
  File "zmq/backend/cython/socket.pyx", line 191, in zmq.backend.cython.socket._recv_copy
  File "zmq/backend/cython/socket.pyx", line 186, in zmq.backend.cython.socket._recv_copy
  File "zmq/backend/cython/checkrc.pxd", line 19, in zmq.backend.cython.checkrc._check_rc
zmq.error.Again: Resource temporarily unavailable

LocalForward 5556.

YUP! Makes sense, because both connections were initiated on the client. Awesome.

Well - I got something...!

WHILE WRITING A DOC - SEE WHAT'S SIMILAR!

Holy COW I can't believe how much better cosine similarity is than a simple dot product (for NLP - image interpretability research uses dot products). Cosine similarity normalizes their lengths, right?

These suggestions are just so awesome.