mirror of
https://github.com/gsi-upm/sitc
synced 2026-05-01 15:14:34 +00:00
Compare commits
3 Commits
59badc1df2
...
d1374320f0
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d1374320f0 | ||
|
|
1e8dbe70a3 | ||
|
|
b3c799e564 |
@@ -239,7 +239,7 @@
|
|||||||
"name": "stderr",
|
"name": "stderr",
|
||||||
"output_type": "stream",
|
"output_type": "stream",
|
||||||
"text": [
|
"text": [
|
||||||
"/home/cif/anaconda3/lib/python3.10/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
|
"--",
|
||||||
" warnings.warn(msg, category=FutureWarning)\n"
|
" warnings.warn(msg, category=FutureWarning)\n"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
@@ -331,7 +331,7 @@
|
|||||||
"source": [
|
"source": [
|
||||||
"vectorizer = CountVectorizer(analyzer=\"word\", stop_words='english', binary=True) \n",
|
"vectorizer = CountVectorizer(analyzer=\"word\", stop_words='english', binary=True) \n",
|
||||||
"vectors = vectorizer.fit_transform(documents)\n",
|
"vectors = vectorizer.fit_transform(documents)\n",
|
||||||
"vectorizer.get_feature_names()"
|
"vectorizer.get_feature_names_out()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -363,9 +363,9 @@
|
|||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"vectorizer = CountVectorizer(analyzer=\"word\", stop_words='english', ngram_range=[2,2]) \n",
|
"vectorizer = CountVectorizer(analyzer=\"word\", stop_words='english', ngram_range=(2,2)) \n",
|
||||||
"vectors = vectorizer.fit_transform(documents)\n",
|
"vectors = vectorizer.fit_transform(documents)\n",
|
||||||
"vectorizer.get_feature_names()"
|
"vectorizer.get_feature_names_out()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -401,7 +401,7 @@
|
|||||||
"\n",
|
"\n",
|
||||||
"vectorizer = TfidfVectorizer(analyzer=\"word\", stop_words='english')\n",
|
"vectorizer = TfidfVectorizer(analyzer=\"word\", stop_words='english')\n",
|
||||||
"vectors = vectorizer.fit_transform(documents)\n",
|
"vectors = vectorizer.fit_transform(documents)\n",
|
||||||
"vectorizer.get_feature_names()"
|
"vectorizer.get_feature_names_out()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -429,9 +429,9 @@
|
|||||||
"train = [doc1, doc2, doc3]\n",
|
"train = [doc1, doc2, doc3]\n",
|
||||||
"vectorizer = TfidfVectorizer(analyzer=\"word\", stop_words='english')\n",
|
"vectorizer = TfidfVectorizer(analyzer=\"word\", stop_words='english')\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# We learn the vocabulary (fit) and tranform the docs into vectors\n",
|
"# We learn the vocabulary (fit) and transform the docs into vectors\n",
|
||||||
"vectors = vectorizer.fit_transform(train)\n",
|
"vectors = vectorizer.fit_transform(train)\n",
|
||||||
"vectorizer.get_feature_names()"
|
"vectorizer.get_feature_names_out()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -51,7 +51,7 @@
|
|||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"In this session we provide a quick overview of the semantic models presented during the classes. In this case, we will use a real corpus so that we can extract meaningful patterns.\n",
|
"In this session, we provide a quick overview of the semantic models presented during the classes. In this case, we will use a real corpus so that we can extract meaningful patterns.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"The main objectives of this session are:\n",
|
"The main objectives of this session are:\n",
|
||||||
"* Understand the models and their differences\n",
|
"* Understand the models and their differences\n",
|
||||||
@@ -69,9 +69,9 @@
|
|||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"We are going to use on of the corpus that come prepackaged with Scikit-learn: the [20 newsgroup datase](http://qwone.com/~jason/20Newsgroups/). The 20 newsgroup dataset contains 20k documents that belong to 20 topics.\n",
|
"We are going to use one of the corpora that come prepackaged with Scikit-learn: the [20 newsgroup dataset](http://qwone.com/~jason/20Newsgroups/). The 20 newsgroup dataset contains 20k documents that belong to 20 topics.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"We inspect now the corpus using the facilities from Scikit-learn, as explain in [scikit-learn](http://scikit-learn.org/stable/datasets/twenty_newsgroups.html#newsgroups)"
|
"We inspect now the corpus using the facilities from Scikit-learn, as explained in [scikit-learn](http://scikit-learn.org/stable/datasets/twenty_newsgroups.html#newsgroups)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -117,19 +117,19 @@
|
|||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Converting Scikit-learn to gensim"
|
"# Converting Scikit-learn to gensim."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Although scikit-learn provides an LDA implementation, it is more popular the package *gensim*, which also provides an LSI implementation, as well as other functionalities. Fortunately, scikit-learn sparse matrices can be used in Gensim using the function *matutils.Sparse2Corpus()*. Anyway, if you are using intensively LDA,it can be convenient to create the corpus with their functions.\n",
|
"Although scikit-learn provides an LDA implementation, it is more popular than the package *gensim*, which also provides an LSI implementation, as well as other functionalities. Fortunately, scikit-learn sparse matrices can be used in Gensim using the function *matutils.Sparse2Corpus()*. Anyway, if you are using intensively LDA,it can be convenient to create the corpus with their functions.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You should install first:\n",
|
"You should install first:\n",
|
||||||
"\n",
|
"\n",
|
||||||
"* *gensim*. Run 'conda install gensim' in a terminal.\n",
|
"* *gensim*. Run 'conda install gensim' in a terminal.\n",
|
||||||
"* *python-Levenshtein*. Run 'conda install python-Levenshtein' in a terminal"
|
"* *python-Levenshtein*. Run 'conda install python-Levenshtein' in a terminal."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -183,7 +183,7 @@
|
|||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Although scikit-learn provides an LDA implementation, it is more popular the package *gensim*, which also provides an LSI implementation, as well as other functionalities. Fortunately, scikit-learn sparse matrices can be used in Gensim using the function *matutils.Sparse2Corpus()*."
|
"Although scikit-learn provides an LDA implementation, it is more popular than the package *gensim*, which also provides an LSI implementation, as well as other functionalities. Fortunately, scikit-learn sparse matrices can be used in Gensim using the function *matutils.Sparse2Corpus()*."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|||||||
Reference in New Issue
Block a user