gensim doc2vec документы не найдены по идентификатору

Question

Вот мой код для обучения мой doc2vec модель

из gensim.models.doc2vec импорта Doc2Vec

from FileDocIterator import FileDocIterator 

doc_file_name = 'doc_6million.txt' 
docs = FileDocIterator(doc_file_name) 
print "Fitting started" 
model = Doc2Vec(docs, size=100, window=5, min_count=5, negative=20, workers=6, iter=4) 
print "Saving model" 
model.save("doc2vec_model") 
print "model saved" 

Теперь давайте взглянем на FileDocIterator

import json 

from gensim.models.doc2vec import TaggedDocument 
from gensim.models import Phrases 

class FileDocIterator(object): 
    def __init__(self, fileName): 
     self.fileName = fileName 
     self.phrase = Phrases.load("phrases") 

    def __iter__(self): 
     for line in open(self.fileName): 
      jsData = json.loads(line) 
      yield TaggedDocument(words=jsData["data"], tags=jsData["id"]) 

Теперь я понимаю, что фразы не используются в этой реализации, но несите меня здесь, давайте посмотрим, как выглядят данные. Вот первая точка данных

{"data":["strategic","and","analytical","technical","program","director","and","innovator","who","inspires","calculated","risk-taking","in","emerging","technologies",",","such","as","cyber","security",",","risk",",","analytics",",","big","data",",","cloud",",","mobility","and","3d","printing",".","known","for","growing","company","profit","through","innovative","thinking","aimed","at","improving","employee","productivity","and","providing","solutions","to","private","industry","and","government","customers",".","recognized","for","invigorating","creative","thinking","and","collaboration","within","large","companies","to","leverage","their","economies","of","scale","to","capture","market","share",".","successful","in","managing","the","risk","and","uncertainty","throughout","the","innovation","lifecycle","by","leveraging","an","innovation","management","framework","to","overcome","barriers",".","track","record","of","producing","results","in","competitive",",","rapidly","changing","environments","where","innovation","and","customer","satisfaction","is","the","business",".","competencies","include",":","innovation","management","cyber",",","risk",",","analytics",",","cloud","computing","and","mobility","technology","development","security","compliance",":","dod/ic","(","nispom",",","icd","503",",","fedramp",")","commercial","(","iso/iec","27002",",","pci","dss",")","relationship","management",":","dod",",","public","sector","and","intelligence","community","change","management","it","security","&","risk","management","(","cissp",")","program",",","product","&","portfolio","management","(","pmp",")","data","analytics","management","(","cchd",")","itil","service", 
"management","(","itilv3-expert",")"], 
"id":"55c37f730d03382935e12767"} 

Я понимаю, что идентификатор, 55c37f730d03382935e12767 должен быть идентификатор документа, тем самым следующий должен дать мне обратно docVector.

model.docvecs["55c37f730d03382935e12767"] 

Вместо этого это то, что выводится.

>>> model.docvecs["55c37f730d03382935e12767"] 
Traceback (most recent call last): 
    File "<stdin>", line 1, in <module> 
    File "/usr/local/lib/python2.7/dist-packages/gensim/models/doc2vec.py", line 341, in __getitem__ 
    return self.doctag_syn0[self._int_index(index)] 
    File "/usr/local/lib/python2.7/dist-packages/gensim/models/doc2vec.py", line 315, in _int_index 
    return self.max_rawint + 1 + self.doctags[index].offset 
KeyError: '55c37f730d03382935e12767' 

Попытка получить наиболее похожими дает следующее назад

>>> model.docvecs.most_similar("55c37f730d03382935e12767") 
Traceback (most recent call last): 
    File "<stdin>", line 1, in <module> 
    File "/usr/local/lib/python2.7/dist-packages/gensim/models/doc2vec.py", line 450, in most_similar 
    raise KeyError("doc '%s' not in trained set" % doc) 
KeyError: "doc '55c37f730d03382935e12767' not in trained set" 

То, что я пытаюсь понять, как векторы Doc сохранен и какие идентификаторы используются. Какая часть моего подхода не работает выше?

Теперь вот что-то интересное, если я сделаю следующее, я вернусь к аналогичным векторам вектора, но для меня это не имеет значения.

>>> model.docvecs.most_similar(str(1)) 
[(u'8', 0.9000369906425476), (u'3', 0.8878246545791626), (u'7', 0.886141836643219), (u'2', 0.8834314942359924), (u'e', 0.8812381029129028), (u'a', 0.8648831248283386), (u'd', 0.8587037920951843), (u'0', 0.8413013219833374), (u'4', 0.8385311365127563), (u'c', 0.8290119767189026)] 

Answer 1

TaggedDocument.tags должен быть список тегов, а не строка. Предоставляя строку, библиотека рассматривает ее как список символов, поэтому одиночные символы интерпретируются как теги документов. Изменить строку:

  yield TaggedDocument(words=jsData["data"], tags=jsData["id"]) 

... к ...

  yield TaggedDocument(words=jsData["data"], tags=[jsData["id"]]) 

... и вы, вероятно, увидеть ожидаемые результаты.