MIT 6.100L Introduction to CS and Programming Using Python, Fall 2022 · Problem set 3, 1) Text to List — text_to_list · Dr. Ana Bell · CC BY-NC-SA 4.0 · MIT publishes no solution for this set; the worked solution is ours
Write the function text_to_list(input_text), which turns the text of a document into a list of its words.
This is the first step of Document Distance, MIT's Problem Set 3. Given two words or documents, you will calculate a score between 0 and 1 that tells you how similar they are. If the words or documents are the same, they get a score of 1; if the documents are completely different, they get a score of 0. You will calculate the score in two different ways and observe whether one works better than the other: the first way uses single word frequencies in the two texts, the second uses the TF-IDF (Term Frequency–Inverse Document Frequency) of words in a file. You do NOT need to worry about case sensitivity anywhere in this problem set: all inputs are lower case.
The first step in any data analysis problem is prepping your data. MIT provides a function called load_file that reads a text file, removes all punctuation, and returns all the text in the file as a string. If the file hello_world.txt reads hello world, hello, then load_file returns 'hello world hello'.
You will further prepare the text by transforming that string into a list representation of the text, where each word is a different element in the list:
print(text_to_list('hello world hello')) # ['hello', 'world', 'hello'] print(text_to_list('hello friends')) # ['hello', 'friends']
You can assume that the only kinds of white space in the text documents will be new lines or space(s) between words (there are no tabs).
Adapted for the browser: files cannot be opened here, so instead of reading MIT's test documents with load_file, the tests hand text_to_list the string load_file would have returned.