Я хотел бы реализовать стандартную процедуру подобия текста с семантическим расстоянием, используя WordNet и NLTK в Python. Это идея: расширить два понятия/прасы/категории A и B с помощью синсетов, гипонимов, гиперним, меронимов, метонимов и вычислить расстояние между двумя сформированными векторами, a и b. Я уверен, как я их вычислим, может быть, как расстояние от косинуса.Основная схожесть текста с помощью синтаксисов WorldNet для сопоставления таксономии/слияния
Мои входные данные для большинства случаев не состоят из фраз, а являются правильными именами или существительными (названия продуктов с маркой или категориями продуктов). Например, я хотел бы определить, что «курорт» - это «роскошный отель» или «черная икра» - это «гурман», «черная икра», «гурман».
В какой степени это может работать и как я могу перемещаться вверх и вниз по WordNet, чтобы сделать его немного более сложным, чем один уровень вверх и вниз с помощью hypo/hyper-nyms.
Я ищу простое базовое решение, которое работает достаточно хорошо, не используя сложные вещи, такие как Whoosh или что-то в этом роде.
Должен ли я использовать что-то лучше, чем WordNet?
UPDATE:
Я обрабатываю каждый словосочетанием следующим образом (с помощью NLTK & WordNet): 1. Для каждого слова в фразе я собирать synset (только существительные), то я дополняю это с синтаксисом гиперним и гипонимов для каждого элемента в синт. На данный момент я захватываю все синтаксисы в список, игнорируя иерархию. 2. Я повторяю процесс для ключевых слов, описывающих каждую из категорий категорий. 3. Теперь у меня есть список синтаксисов для каждой категории и для моей цели. Просто вычислите расстояние до каждого (косинус или расстояние Ву и Палмера). Я собираю попарные расстояния в моих двух векторах, суммируя их, нормализуясь количеством ключевых слов, описывающих категорию или цель. Затем я выбираю минимальное расстояние.
Звучит как довольно простой и неэффективный. Каков следующий шаг, чтобы сделать его лучше?
Мне интересно это сделать с нуля, это также лучшее упражнение, чтобы понять, как все работает и как это нужно делать.
Пример: word_list - цель: [ 'школа', 'дети', 'учитель']
категории: [[ 'бизнес', 'организация', 'Компания'], ['education', 'school', 'university']]
расширенный список для целевой концепции 'education', 3 keywords: [Synset ('school.n.01'), Synset ('school.n. 02 '), Synset (' school.n.03 '), Synset (' school.n.04 '), Synset (' school.n.05 '), Synset (' school.n.06 '), Synset ('school.n.07'), Synset ('child.n.01'), Synset ('kid.n.02'), S ynset ('kyd.n.01'), Synset ('child.n.02'), Synset ('kid.n.05'), Synset ('teacher.n.01'), Synset ('teacher.n .02 '), Synset (' education_institution.n.01 '), Synset (' building.n.01 '), Synset (' education.n.03 '), Synset (' body.n.02 '), Synset ('time_period.n.01'), Synset ('education_institution.n.01'), Synset ('animal_group.n.01'), Synset ('academy.n.03'), Synset ('alma_mater.n. 01 '), Synset (' conservatory.n.01 '), Synset (' matching_school.n.01 '), Synset (' crammer.n.03 '), Synset (' dance_school.n.01 '), Synset ('dancing_school.n.01'), Synset ('day_school.n.02'), Synset ('direct-grant_school.n.01'), Synset ('driving_school.n.01'), Synset ('finish_school.n .01 '), Synset (' flying_school.n.01 '), Synset (' grade_school.n.01 '), Synset (' graduate_school.n.01 '), Synset (' language_school.n.01 '), Synset (' night_school.n.01 '), Synset (' nursing_school.n.01 '), Synset (' private_school.n.01 '), Synset ('public_school.n.01'), Synset ('relig_school.n.01'), Synset ('riding_school.n.01'), Synset ('secondary_school.n.01'), Synset ('secretarial_school.n.01 '), Synset (' sunday_school.n.01 '), Synset (' technical_school.n.01 '), Synset (' training_school.n.01 '), Synset (' veterinary_school.n.01 '), Synset (' conservation.n.02 '), Synset (' day_school.n.03 '), Synset (' art_nouveau.n.01 '), Synset (' ashcan_school.n.01 '), Synset (' deconstructivism.n.01 '), Synset ('history_school.n.01'), Synset ('lake_poets.n.01'), Synset ('pointillism.n.01'), Synset ('secession.n.01')]
Расширенный список для категории «бизнес», 3 ключевых слова, 223 в расширенном списке: [Synset ('business.n.01'), Synset ('commercial_enterprise.n.02'), Synset ('occupation.n .01 '), Synset (' business.n.04 '), Synset (' business.n.05 '), Synset (' business.n.06 '), Synset (' business.n.07 '), Synset ('clientele.n.01'), Synset ('business.n.09'), Synset ('organization.n.01'), Synset ('assembly.n.03'), Synset ('Administration.n. 02 '), Synset (' organization.n.04 '), Synset (' organization.n.05 '), Synset (' organization.n.06 '), Synset (' Constitution.n.02 '), Synset ('company.n.01'), Synset ('company.n.02'), Synset ('company.n.03'), Synset ('company.n.04'), Synset ('caller.n.01 '), Synset (' company.n.06 '), Synset (' party.n.03 '), Synset (' ship's_company.n.01 '), Synset (' company.n.09 '), Synset ('enterprise.n.02'), Synset ('commerce.n.01'), Synset ('activity.n.01'), Synset ('concern.n.04'), Synset ('aim.n. 02 '), Synset (' business_activity.n.01 '), Synset (' sector.n.02 '), Synset (' people.n.01 '), Synset (' act.n.01 '), Synset ('social_group.n.01'), Synset ('structure.n.03'), Synset ('body.n.02'), Synset ('Administration.n.01'), Synset ('orderliness.n.01 '), Synset (' activity.n.01 '), Synset (' begin.n.05 '), Synset (' Instance ('organization.n.01'), Synset ('organization.n.01'), Synset ('youth.n.01'), Synset ('organization.n.01'), Synset ('visitor.n.01'), Synset ('social_gathering.n.01'), Synset ('set.n.05'), Synset ('дополнение.n.03'), Synset ('unit.n.03'), Synset ('агентство .n.02 '), Synset (' brokerage.n.02 '), Synset (' carrier.n.05 '), Synset (' chain.n.04 '), Synset (' firm.n.01 '), , Synset ('franchise.n.02'), Synset ('manufacturer.n.01'), Synset ('partner.n.01'), Synset ('processor.n.01'), Synset ('shipbuilder. n.03 '), Synset (' underperformer.n.02 '), Synset (' advertising.n.02 '), Synset (' agribusiness.n.01 '), Synset (' butchery.n.02 '), Synset ('construction.n.07'), Synset ('discount_business.n.01'), Synset ('employee-owned_enterprise.n.01'), Synset ('field.n.06'), Synset ('finance .n.01 '), Synset (' fishing.n.02 '), Synset (' industry.n.02 '), Synset (' packaging.n.01 '), Synset (' printing.n.02 ') , Synset ('publication.n.04'), Synset ('real-estate_business.n.01'), Synset ('storage.n.03'), Synset ('tourism.n.01'), Synset (' transport.n.05 '), Synset (' vent ure.n.03 '), Synset (' accountancy.n.01 '), Synset (' meeting.n.05 '), Synset (' career.n.01 '), Synset (' catering.n.01 '), Synset ('confectionery.n.03'), Synset ('employment.n.02'), Synset ('farming.n.02'), Synset ('game.n.10'), Synset ('metier .n.02 '), Synset (' photography.n.03 '), Synset (' position.n.06 '), Synset (' profession.n.02 '), Synset (' sport.n.02 '), , Synset ('trade.n.02'), Synset ('treadmill.n.03'), Synset ('occasionions.n.01'), Synset ('land-office_business.n.01'), Synset (' trade.n.03 '), Synset (' big_business.n.01 '), Synset (' shtik.n.02 '), Synset (' adhocracy.n.01 '), Synset (' affiliate.n.02 '), Synset ('alliance.n.03'), Synset ('association.n.01'), Synset ('blue.n.03'), Synset ('bureaucracy.n.03'), Synset ('компания .n.04 '), Synset (' defense.n.09 '), Synset (' deputation.n.01 '), Synset (' enterprise.n.02 '), Synset (' creation.n.05 '), , Synset ('federation.n.01'), Synset ('fiefdom.n.02'), Synset ('fire_brigade.n.01'), Synset ('force.n.04'), Synset ('girl_scouts. n.01 '), Synset (' grey.n.04 '), Synset (' hierarchy.n.02 '), Synset (' ho st.n.06 '), Synset (' instit.n.01 '), Synset (' line_of_defense.n.01 '), Synset (' line_organization.n.01 '), Synset (' machine.n.03 '), Synset ('machine.n.05'), Synset ('music_organization.n.01'), Synset ('негосударственный_организация.n.01'), Synset ('party.n.01'), Synset ('peace_corps .n.01 '), Synset (' polity.n.02 '), Synset (' pool.n.03 '), Synset (' professional_organization.n.01 '), Synset (' quango.n.01 '), , Synset ('tammany_hall.n.01'), Synset ('union.n.01'), Synset ('unit.n.03'), Synset ('calendar.n.01'), Synset ('classic_system. n.01 '), Synset (' contrivance.n.04 '), Synset (' coord_system.n.01 '), Synset (' data_structure.n.01 '), Synset (' design.n.02 '), Synset ('distribution.n.01'), Synset ('gene_map.n.01'), Synset ('kinship_system.n.01'), Synset ('lattice.n.01'), Synset ('living_arrangement.n .01 '), Synset (' ontology.n.01 '), Synset (' county_council.n.01 '), Synset (' curia.n.01 '), Synset (' executive.n.02 '), Synset ('government_officials.n.01'), Synset ('judiciary.n.01'), Synset ('management.n.02 '), Synset (' top_brass.n.01 '), Synset (' nonprofit_organization.n.01 '), Synset (' rationalization.n.04 '), Synset (' reorganization.n.01 '), Synset ('self-organization.n.01'), Synset ('syndication.n.01'), Synset ('listing.n.02'), Synset ('order.n.15'), Synset ('randomization.n .01 '), Synset (' systematization.n.01 '), Synset (' территориализация.n.01 '), Synset (' collectivization.n.01 '), Synset (' colonization.n.01 '), Synset ('communization.n.02'), Synset ('federation.n.03'), Synset ('unionization.n.01'), Synset ('broadcasting_company.n.01'), Synset ('bureau_de_change.n. 01 '), Synset (' car_company.n.01 '), Synset (' closed_shop.n.01 '), Synset (' Corporate_investor.n.01 '), Synset (' distributor.n.03 '), Synset ('dot-com.n.01'), Synset ('drug_company.n.01'), Synset ('east_india_company.n.01'), Synset ('electronics_company.n.01'), Synset ('film_company.n .01 '), Synset (' food_company.n.01 '), Synset (' furniture_company.n.01 '), Synset (' holding_company.n.01 '), Synset (' joint-stock_company.n.01 '), , Synset ('limited_company.n.01'), Synset (' livery_company.n.01 '), Synset (' mining_company.n.01 '), Synset (' mover.n.04 '), Synset (' oil_company.n.01 '), Synset (' open_shop.n.01 '), Synset ('packaging_company.n.01'), Synset ('pipe_company.n.01'), Synset ('printing_concern.n.01'), Synset ('record_company.n.01'), Synset ('service .n.04 '), Synset (' shipper.n.02 '), Synset (' shipping_company.n.01 '), Synset (' steel_company.n.01 '), Synset (' stock_company.n.01 '), , Synset ('secondary_company.n.01'), Synset ('target_company.n.01'), Synset ('think_tank.n.01'), Synset ('transport_company.n.01'), Synset ('union_shop. n.01 '), Synset (' white_knight.n.01 '), Synset (' trainband.n.01 '), Synset (' freemasonry.n.01 '), Synset (' ballet_company.n.01 '), Synset ('chorus.n.05'), Synset ('circus.n.01'), Synset ('minstrel_show.n.01'), Synset ('minstrelsy.n.01'), Synset ('opera_company.n .01 '), Synset (' theater_company.n.01 '), Synset (' attendance.n.03 '), Synset (' cohort.n.01 '), Synset (' number.n.07 '), Synset ('fatigue_party.n.01'), Synset ('landing_party.n.01'), Synset ('party_to_the_action .n.01 '), Synset (' rescue_party.n.01 '), Synset (' search_party.n.01 '), Synset (' stretcher_party.n.01 '), Synset (' war_party.n.01 '), ]
расширенный список для категории понятие 'образование' - 97 synsets: [Synset ('education.n.01'), Synset ('education.n.02'), Synset ('education.n.03'), Synset ('education.n.04'), Synset ('education.n.05'), Synset ('department_of_education.n.01'), Synset ('school.n.01'), Synset ('school .n.02 '), Synset (' school.n.03 '), Synset (' school.n.04 '), Synset (' school.n.05 '), Synset (' school.n.06 '), , Synset ('school.n.07'), Synset ('university.n.01'), Synset ('university.n.02'), Synset ('university.n.03'), Synset ('activity. n.01 '), Synset (' content.n.05 '), Synset (' learning.n.01 '), Synset (' profession.n.02 '), Synset (' upbringing.n.01 '), Synset ('executive_department.n.01'), Synset ('education_institution.n.01'), Synset ('building.n.01'), Synset ('education .n.03 '), Synset (' body.n.02 '), Synset (' time_period.n.01 '), Synset (' education_institution.n.01 '), Synset (' animal_group.n.01 '), , Synset ('body.n.02'), Synset ('creation.n.04'), Synset ('education_institution.n.01'), Synset ('coeducation.n.01'), Synset ('continue_education. n.01 '), Synset (' course.n.01 '), Synset (' elementary_education.n.01 '), Synset (' extension.n.04 '), Synset (' extracurricular_activity.n.01 '), Synset ('higher_education.n.01'), Synset ('secondary_education.n.01'), Synset ('team_teaching.n.01'), Synset ('work-study_program.n.01'), Synset ('просвещение .n.01 '), Synset (' eruditeness.n.01 '), Synset (' experience.n.01 '), Synset (' foundation.n.04 '), Synset (' physical_education.n.01 '), , Synset ('acculturation.n.03'), Synset ('mastering.n.01'), Synset ('school.n.03'), Synset ('self-education.n.01'), Synset (' special_education.n.01 '), Synset (' vocational_training.n.01 '), Synset (' teaching.n.01 '), Synset (' academy.n.03 '), Synset (' alma_mater.n.01 '), Synset ('conservatory.n.01'), Synset ('matching_s chronic.n.01 '), Synset (' crammer.n.03 '), Synset (' dance_school.n.01 '), Synset (' dancing_school.n.01 '), Synset (' day_school.n.02 '), Synset ('direct-grant_school.n.01'), Synset ('driving_school.n.01'), Synset ('finish_school.n.01'), Synset ('flying_school.n.01'), Synset ('grade_school.n.01'), Synset ('graduate_school.n.01'), Synset ('language_school.n.01'), Synset ('night_school.n.01'), Synset ('nursing_school.n.01 '), Synset (' private_school.n.01 '), Synset (' public_school.n.01 '), Synset (' relig_school.n.01 '), Synset (' riding_school.n.01 '), Synset (' secondary_school.n.01 '), Synset (' secretarial_school.n.01 '), Synset (' sunday_school.n.01 '), Synset (' technical_school.n.01 '), Synset (' training_school.n.01 '), Synset ('veterinary_school.n.01'), Synset ('conservatory.n.02'), Synset ('day_school.n.03'), Synset ('art_nouveau.n.01'), Synset ('ashcan_school .n.01 '), Synset (' deconstructivism.n.01 '), Synset (' history_school.n.01 '), Synset (' lake_poets.n.01 '), Synset (' pointillism.n.01 '), , Synset ('secession.n.01 '), Synset (' gown.n.02 '), Synset (' varsity.n.01 '), Synset (' city_university.n.01 '), Synset (' oxbridge.n.01 '), Synset ('redbrick_university.n.01'), Synset ('multiversity.n.01'), Synset ('open_university.n.01')]
Расширенный список для моей цели, 57 synsets: [Synset ('school .n.01 '), Synset (' school.n.02 '), Synset (' school.n.03 '), Synset (' school.n.04 '), Synset (' school.n.05 '), , Synset ('school.n.06'), Synset ('school.n.07'), Synset ('child.n.01'), Synset ('kid.n.02'), Synset ('kyd. n.01 '), Synset (' child.n.02 '), Synset (' kid.n.05 '), Synset (' teacher.n.01 '), Synset (' teacher.n.02 '), Synset ('education_institution.n.01'), Synset ('building.n.01'), Synset ('education.n.03'), Synset ('body.n.02'), Synset ('time_period.n .01 '), Synset (' education_institution.n.01 '), Synset (' animal_group.n.01 '), Synset (' academy.n.03 '), Synset (' alma_mater.n.01 '), Synset ('conservatory.n.01'), Synset ('match_school.n.01'), Synset ('crammer.n.03'), Synset ('dance_school.n.01'), Synset ('dancing_school.n.01'), Synset ('day_school .n.02 '), Synset (' direct-grant_school.n.01 '), Synset (' driving_school.n.01 '), Synset (' finish_school.n.01 '), Synset (' flying_school.n.01 '), Synset (' grade_school.n.01 '), Synset (' graduate_school.n.01 '), Synset (' language_school.n.01 '), Synset (' night_school.n.01 '), Synset (' (saint_school.n.01 '), Synset (' public_school.n.01 '), Synset (' relig_school.n.01 '), Synset (' riding_school.n.01 '), Synset), Synset ('secondary_school.n.01'), Synset ('secretarial_school.n.01'), Synset ('sunday_school.n.01'), Synset ('technical_school.n.01'), Synset ('training_school .n.01 '), Synset (' veterinary_school.n.01 '), Synset (' conservatory.n.02 '), Synset (' day_school.n.03 '), Synset (' art_nouveau.n.01 '), , Synset ('ashcan_school.n.01'), Synset ('deconstructivism.n.01'), Synset ('history_school.n.01'), Synset ('lake_poets.n.01'), Synset ('pointillism. n.01 '), Synset (' secession.n.01')]
У меня есть 3 векторов, целевых - 57, бизнес - 223, и образование - 97.
Теперь вычисляют попарные расстояния В и Палмер между мишенью и бизнесом, разделяйте по 57x223 = 12711; между мишенью и образованием, разделите на 57x97 = 5529.
цель для бизнеса расстояние: 2305,709117171037/5529 = 0,9125370052417936 цель на расстоянии образования: +5045,417101981877/12711 = 0,39693313680921066
Минимальное расстояние до образования. Это правильный ответ.