Как разбить строку на запятую или периоды в nltk

Question

Я хочу разделить строку на запятую и/или периоды в nltk. Я пробовал с sent_tokenize(), но он отделяется только от периодов.

Я также попробовал этот код

from nltk.tokenize.punkt import PunktSentenceTokenizer, PunktLanguageVars 
ex_sent = "This is an example showing sentence filtration.This is how it is done, in case of Python I want to learn more. So, that i can have some experience over it, by it I mean python." 
class CommaPoint(PunktLanguageVars): 
    sent_end_chars = ('.','?','!',',') 
tokenizer = PunktSentenceTokenizer(lang_vars = CommaPoint()) 
n_w=tokenizer.tokenize(ex_sent) 
print n_w 

Выход для кода выше является

['This is an example showing sentence filtration.This is how it is done,' 'in case of Python I want to learn more.' 'So,' 'that i can have some experience over it,' 'by it I mean python.\n'] 

Когда я пытаюсь дать ". без какого-либо пространства, она принимает его как слово

Я хочу, чтобы выход в

['This is an example showing sentence filtration.' 'This is how it is done,' 'in case of Python I want to learn more.' 'So,' 'that i can have some experience over it,' 'by it I mean python.'] 

Answer 1

Как о чем-то более простом с re:

>>> import re 
>>> sent = "This is an example showing sentence filtration.This is how it is done, in case of Python I want to learn more. So, that i can have some experience over it, by it I mean python." 
>>> re.split(r'[.,]', sent) 
['This is an example showing sentence filtration', 'This is how it is done', ' in case of Python I want to learn more', ' So', ' that i can have some experience over it', ' by it I mean python', ''] 

Чтобы сохранить разделитель, вы можете использовать группу:

>>> re.split(r'([.,])', sent) 
['This is an example showing sentence filtration', '.', 'This is how it is done', ',', ' in case of Python I want to learn more', '.', ' So', ',', ' that i can have some experience over it', ',', ' by it I mean python', '.', '']