--- title: "Introduction to inpdfr package" author: "François Rebaudo, Institut de Recherche pour le Développement, UMR EGCE, Univ.Paris Sud-CNRS-IRD-Univ.Paris Saclay, France" date: "`r Sys.Date()`" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Introduction to inpdfr package} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- The `inpdfr` package is primarily designed for analysing and comparing PDF and/or TXT documents. For this Vignette we used PDF articles from the Journal of Statistical Software available at https://www.jstatsoft.org. Specifically we used the 10 articles from volume 68 (2015), which can be freely downloaded: - v68i01.pdf; CovSel: An R Package for Covariate Selection When Estimating Average Causal Effects, by Jenny Häggström, Emma Persson, Ingeborg Waernbaum, Xavier de Luna. - v68i02.pdf; sms: An R Package for the Construction of Microdata for Geographical Analysis, by Dimitris Kavroudakis. - v68i03.pdf; Parallel Sequential Monte Carlo for Efficient Density Combination: The DeCo MATLAB Toolbox, by Roberto Casarin, Stefano Grassi, Francesco Ravazzolo, Herman K. van Dijk. - v68i04.pdf; Bayesian Model Averaging Employing Fixed and Flexible Priors: The BMS Package for R, by Stefan Zeugner, Martin Feldkircher. - v68i05.pdf; Bayesian Model Averaging and Jointness Measures for gretl, by Marcin Błażejowski, Jacek Kwiatkowski. - v68i06.pdf; Visually Exploring Missing Values in Multivariable Data Using a Graphical User Interface, by Xiaoyue Cheng, Dianne Cook, Heike Hofmann. - v68i07.pdf; equateIRT: An R Package for IRT Test Equating, by Michela Battauz. - v68i08.pdf; A SAS Program Combining R Functionalities to Implement Pattern-Mixture Models, by Pierre Bunouf, Geert Molenberghs, Jean-Marie Grouin, Herbert Thijs. - v68i09.pdf; POPS: A Software for Prediction of Population Genetic Structure Using Latent Regression Models, by Flora Jay, Olivier François, Eric Y. Durand, Michael G. B. Blum. - v68i10.pdf; Semi-Parametric Maximum Likelihood Method for Interaction in Case-Mother Control-Mother Designs: Package SPmlficmcm, by Moliere Nguile-Makao, Alexandre Bureau. We used these files for the purpose of this vignette, but I encourage you to test `inpdfr` package with your own publications, or those of your lab and colleagues. The package uses XPDF (http://www.xpdfreader.com/download.html) for PDF to text extraction. You need to install XPDF before using `inpdfr` package. Depending on your operating system, you may need to restart your computer after installing XPDF. If you do not want to use XPDF, you can extract the content of your PDF files with the method of your choice and then store the content in TXT files. The only function making use of XPDF is `getPDF` which can be substituted with the `getTXT` function. ## 1. Using inpdfr from command line ### 1.1 Obtaining the word-occurrence data.frame from a set of documents #### 1.1.1. Extracting text from PDF To extract text from PDF files, you need to specify the directory where your files are located: mywd <- "/home/user/myWD/JSS/" Then list your PDF files using `getListFiles` function: listFilesExt <- getListFiles(mywd) #> $pdf #> [1] "v68i01.pdf" "v68i02.pdf" "v68i03.pdf" "v68i04.pdf" #> [5] "v68i05.pdf" "v68i06.pdf" "v68i07.pdf" "v68i08.pdf" #> [9] "v68i09.pdf" "v68i10.pdf" #> #> $txt #> NULL To extract text from PDF files, use the `getPDF` function: wordFreqPDF <- getPDF(myPDFs = listFilesExt$pdf) #> [[1]]$wc #> freq stem word #> 626 264 the the #> 32 141 and and #> ... ... ... ... #> #> [[1]]$name #> [1] "v68i01" #> #> ... You will get a list where each element corresponds to a list composed of a data.frame (freq = word frequency; stem = stem word; word = word) and the name of the original PDF file without the extension. If you also have TXT files, you can use the `getTXT` function which works similarly. To merge the results of the PDF and TXT extraction, use the `append` function as shown bellow: wordFreqPDF <- getPDF(myPDFs = listFilesExt$pdf) wordFreqTXT <- getTXT(myTXTs = listFilesExt$txt) wordFreq <- append(wordFreqPDF, wordFreqTXT) #### 1.1.2. Excluding stop words In order to exclude stop words, use the `excludeStopWords` function which takes the list previously created and the language as arguments: wordFreq <- excludeStopWords(wordF = wordFreq, lang = "English") #> [[1]]$wc #> freq stem word #> 135 101 covari covariates #> 144 46 data data #> ... ... ... ... #> #> [[1]]$name #> [1] "v68i01" #> #> ... In our case, "the" and "and" where supressed from the data.frame. #### 1.1.3. Truncation of the number of words Optionally, you can truncate the number of words in each data.frame using the `truncNumWords` function: wordFreq <- truncNumWords(maxWords = Inf, wordF = wordFreq) Specifying `maxWords = Inf` won't truncate the data.frames. #### 1.1.4. Merging data.frames To obtain a word occurence data.frame, each element of the wordFreq list must be merged. This operation is performed with the `mergeWordFreq` function: mergedD <- mergeWordFreq(wordF = wordFreq) #> word v68i01 v68i02 v68i03 v68i04 v68i05 ... #> stem2076 model 11 25 89 420 253 ... #> stem722 data 46 95 11 43 18 ... #> ... #### 1.1.5. Quick function All theses tasks can be performed with the `getwordOccuDF` function which takes the working directory and the language as arguments: mergedD <- getwordOccuDF(mywd = "/home/user/myWD/JSS/", language = "English") ### 1.2. Computing a set of analysis from the word-occurrence data.frame A folder named "RESULTS" is created in your working directory and contains the output files for each analysis performed. #### 1.2.1. Simple manipulations of the word occurrence data.frame Simple manipulations can be easily performed from the word occurrence data.frame. The number of words (excluding stop words) can be computed as following: numWords <- apply(mergedD[,2:ncol(mergedD)], MARGIN = 2, FUN = sum) Or the number of unique words: numUniqueWords <- apply(mergedD[,2:ncol(mergedD)], MARGIN = 2, FUN = function(i) {length(i[i > 0])}) Considering the number of words as an "area", and the number of unique words as "species", we can easily build a "species-area relationships" analysis (which is commonly Log-Log transformed): plot(x = log(numWords), y = log(numUniqueWords), pch = 16) text(x = log(numWords), y = log(numUniqueWords), labels=names(mergedD[,2:ncol(mergedD)]), cex= 0.7,pos = 3) lmSAR <- lm(log(numUniqueWords) ~ log(numWords)) summary(lmSAR) abline(lmSAR) #> Call: #> lm(formula = log(numUniqueWords) ~ log(numWords)) #> #> Residuals: #> Min 1Q Median 3Q Max #> -0.31186 -0.03016 0.02944 0.05401 0.15724 #> #> Coefficients: #> Estimate Std. Error t value Pr(>|t|) #> (Intercept) 1.4693 1.2606 1.166 0.27738 #> log(numWords) 0.6246 0.1524 4.099 0.00344 ** #> --- #> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 #> #> Residual standard error: 0.1316 on 8 degrees of freedom #> Multiple R-squared: 0.6775, Adjusted R-squared: 0.6372 #> F-statistic: 16.8 on 1 and 8 DF, p-value: 0.003441 The slope is significantly different from zero, so that longer articles have a higher specific richness in terms of number of unique words. Additional analyses from this perspective can be found in numerous R ecological packages. A good place to start is the `vegan` package. #### 1.2.2. Word cloud Assuming your word occurrence data.frame is named "mergedD", you can compute word clouds with the `makeWordcloud` function. The getPlot argument controls which word clouds should be computed. If getPlot[1] == TRUE, then a word cloud is made for each file. If getPlot[2] == TRUE, then a word cloud is made for the set of documents. makeWordcloud(wordF = mergedD, wcminFreq = 50, wcmaxWords = Inf, wcRandOrder = FALSE, wcCol = RColorBrewer::brewer.pal(8,"Dark2"), getPlot = c(FALSE,TRUE)) #### 1.2.3. Summary statistics The function `getSummaryStatsBARPLOT` allows you to compute a barplot of the number of unique words per document. It returns the number of unique words per document. getSummaryStatsBARPLOT(wordF = mergedD) #> [1] 597 800 969 1019 838 788 541 822 823 568 The function `getSummaryStatsHISTO` allows you to compute an histogram of the number of word per document. getSummaryStatsHISTO(wordF = mergedD) The function `getSummaryStatsOCCUR` allows you to compute a scatter plot with the proportion of documents using similar words, and returns the corresponding table. getSummaryStatsOCCUR(wordF = mergedD) #> dfTableP[, 3] dfTableP[, 2] #> 1 (0,0.1] 2296 #> 2 (0.1,0.2] 467 #> 3 (0.2,0.3] 217 #> 4 (0.3,0.4] 142 #> 5 (0.4,0.5] 128 #> 6 (0.5,0.6] 101 #> 7 (0.6,0.7] 58 #> 8 (0.7,0.8] 67 #> 9 (0.8,0.9] 62 #> 10 (0.9,1] 57 In our example, we can see that 57 words were used in 90 to 100% of the articles, or that 58 words were used in 60 to 70% of the articles, while 2296 words were specific to an article. Comparison with a special issue should give a different repartition with more words common to all articles. #### 1.2.4. Word frequency The function `getMostFreqWord` returns the most frequent words in the word occurrence data.frame. It also compute a scatter plot with the frequency of each word for each document. getMostFreqWord(wordF = mergedD, numWords = 10) #> [1] "model" "data" "prior" "package" "variable" #> [6] "test" "values" "estimate" "statistical" "set" You may want to normalize the frequency by the number of words in each document. This can be easily done (in our example, the most frequent words are the same, but the corresponding scatter plots differ): mergedDNorm <- data.frame(word = as.character(mergedD[,1]), t(t(mergedD[,2:ncol(mergedD)]) / apply(mergedD[,2:ncol(mergedD)], MARGIN=2, FUN=sum)) * 100) getMostFreqWord(wordF = mergedDNorm, numWords = 10) #> [1] "model" "data" "prior" "package" "variable" #> [6] "test" "values" "estimate" "statistical" "set" The function `getMostFreqWordCor` compute the correlation between most frequent words. Images of the correlation matrices are also provided in the "RESULTS" folder. In our set of PDFs, we can see for example that "model" is significantly correlated with "prior", or that "statistical" is significantly correlated with "varaible": getMostFreqWordCor(wordF = mergedD, numWords = 10) #> $cor #> model data prior package variable ... #> model 1.0000000 -0.3835130 0.9564451 0.194231566 0.2746420 ... #> data -0.3835130 1.0000000 -0.2039196 0.108141861 0.5050901 ... #> prior 0.9564451 -0.2039196 1.0000000 0.188332000 0.2856416 ... #> package 0.1942316 0.1081419 0.1883320 1.000000000 0.4097255 ... #> variable 0.2746420 0.5050901 0.2856416 0.409725475 1.0000000 ... #> test -0.1517949 -0.3209755 -0.1651302 0.278446147 -0.3004002 ... #> values -0.3621747 0.5553868 -0.3198548 -0.232902022 0.5250301 ... #> estimate -0.1736209 -0.3753552 -0.1576511 -0.009370034 -0.5549054 ... #> statistical 0.4792956 0.2101346 0.5621689 0.183642865 0.6776569 ... #> set 0.2755856 -0.2963624 0.2279321 -0.130621530 0.2758461 ... #> #> $pval #> model data prior package variable ... #> model 0.000000e+00 0.27394969 1.493632e-05 0.5907879 0.44252229 ... #> data 2.739497e-01 0.00000000 5.720168e-01 0.7661868 0.13646367 ... #> prior 1.493632e-05 0.57201681 0.000000e+00 0.6023278 0.42369314 ... #> package 5.907879e-01 0.76618681 6.023278e-01 0.0000000 0.23963804 ... #> variable 4.425223e-01 0.13646367 4.236931e-01 0.2396380 0.00000000 ... #> test 6.754944e-01 0.36584180 6.484673e-01 0.4359677 0.39903200 ... #> values 3.037403e-01 0.09557406 3.676131e-01 0.5172745 0.11916335 ... #> estimate 6.314473e-01 0.28514377 6.635822e-01 0.9795048 0.09592275 ... #> statistical 1.610148e-01 0.56009587 9.074744e-02 0.6115571 0.03130403 ... #> set 4.408923e-01 0.40570937 5.265050e-01 0.7190909 0.44044280 ... The function `getXFreqWord` returns the words which have been found at leat X times in the set of documents. getXFreqWord(wordF = mergedD, occuWords = 200) #> [1] "model" "data" "prior" #> [4] "package" "variable" "test" #> [7] "values" "estimate" "statistical" #> [10] "set" "miss" "parameter" #> [13] "covariance" "coefficient" "imputation" #> [16] "number" "journal" "equating" #> [19] "function" "results" "method" #> [22] "software" "average" #### 1.2.5. Correspondance analysis The function `doCA` performs a correspondance analysis on the basis of the word occurrence data.frame, with the associated plot. doCA(wordF = mergedD) #> Principal inertias (eigenvalues): #> 1 2 3 4 5 6 7 ... #> Value 0.500619 0.472982 0.442737 0.411129 0.401536 0.389836 0.374116 ... #> Percentage 13.83% 13.07% 12.24% 11.36% 11.1% 10.77% 10.34% ... #> #> Rows: #> ... #> #> Columns: #> v68i01 v68i02 v68i03 v68i04 v68i05 v68i06 ... #> Mass 0.062176 0.095228 0.126130 0.165061 0.104936 0.097500 ... #> ChiDist 2.405088 2.046214 1.685774 1.438954 1.636650 1.864347 ... #> Inertia 0.359653 0.398717 0.358441 0.341773 0.281083 0.338890 ... #> Dim. 1 -0.011166 0.768156 0.513466 0.475236 0.383365 0.238685 ... #> Dim. 2 0.794833 1.180390 -0.146855 -1.655962 -1.109124 0.946243 ... #### 1.2.6. Cluster analysis The function `doCluster` performs a cluster analysis with the associated dendrogram. doCluster(wordF = mergedD, myMethod = "ward.D2", gp = FALSE, nbGp = 3) #> Call: #> stats::hclust(...) #> #> Cluster method : ward.D2 #> Distance : euclidean #> Number of objects: 10 #### 1.2.7. K-means cluster analysis The function `doKmeansClust` performs a k-means cluster analysis with the associated cluster plot. doKmeansClust(wordF = mergedD, nbClust = 4, nbIter = 10, algo = "Hartigan-Wong") #> K-means clustering with 4 clusters of sizes 1, 2, 6, 1 #> #> Cluster means: #> v68i01 v68i02 v68i03 v68i04 v68i05 v68i06 v68i07 ... #> 1 501.5376 549.1102 589.0849 738.2107 569.1072 593.9343 0.0000 ... #> 2 392.8950 422.6009 221.9471 526.7186 221.9471 484.8954 579.0960 ... #> 3 257.0480 290.6130 431.3375 632.9743 418.1930 320.8966 536.3371 ... #> 4 622.2564 641.1404 632.0633 0.0000 421.3739 680.6240 738.2107 ... #> #> Clustering vector: #> v68i01 v68i02 v68i03 v68i04 v68i05 v68i06 v68i07 v68i08 v68i09 v68i10 #> 3 3 2 4 2 3 1 3 3 3 #> #> Within cluster sum of squares by cluster: #> [1] 0.0 220295.9 672154.6 0.0 #> (between_SS / total_SS = 67.2 %) #> #> Available components: #> #> [1] "cluster" "centers" "totss" "withinss" ... #### 1.2.8. Metacommunity analysis with entropart Discussing the analyses performed here are out of the scope of this vignette. Briefly, the function `doMetacomEntropart` uses the `entropart` package and the functions `DivEst`, `DivPart`, `DivProfile`, and `MetaCommunity`. Results are provided as plots or TXT files in the "RESULTS" folder. Words are considered as species, word occurrences as abundances, and documents as communities. doMetacomEntropart(wordF = mergedD) #> Meta-community (class 'MetaCommunity') made of 25170 individuals in 10 #> communities and 3595 species. #> #> Its sample coverage is 0.973223513822329 #> #> Community weights are: #> v68i01 v68i02 v68i03 v68i04 v68i05 v68i06 v68i07 v68i08 v68i09 v68i10 #> 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 #> Community sample numbers of individuals are: #> v68i01 v68i02 v68i03 v68i04 v68i05 v68i06 v68i07 v68i08 v68i09 v68i10 #> 2517 3855 5106 6682 4248 3947 3723 4334 3439 2631 #> Community sample coverages are: #> v68i01 v68i02 v68i03 v68i04 v68i05 v68i06 v68i07 ... #> 0.9070729 0.9250581 0.9265726 0.9440365 0.9268099 0.9176794 0.9444160 ... #> #> ... #### 1.2.9. Metacommunity analysis with metacom Discussing the analyses performed here are out of the scope of this vignette. Briefly, the function `doMetacomMetacom` uses the `metacom` package and the `metacom` function. Just like before, words are considered as species, word occurrences as abundances, and documents as communities (allowing the metacommunity analysis). doMetacomMetacom(wordF = mergedD, numSim = 10, limit = "Inf") #> [1] "Identified community structure: Random" #> $Comm #> ... #> #> $Coherence #> output #> embedded absences 20251 #> z 1.91875604260688 #> pval 0.0550152153018832 #> sim.mean 22109.1 #> sim.sd 968.387829791809 #> method r1 #> #> $Turnover #> output #> replacements 5961653 #> z -2.67166401618644 #> pval 0.00754761772110864 #> sim.mean 3470121.1 #> sim.sd 932576.80790133 #> method r1 #> #> $Boundary #> index P df #> 1 0 0.4234722 3592 #### 1.2.10. Quick function All theses tasks can be performed with the `getAllAnalysis` function which takes the word-occurrence data.frame as argument: getAllAnalysis(dataset = mergedD) ## 2. Using inpdfr from the Graphical User Interface (GUI) To load the RGtk2 GUI, use the function `loadGUI` available only in the gitHub webpage (https://github.com/frareb/inpdfr): loadGUI() All function used to build the GUI were made available so that any developer can easily access its content. They are not intended to be used by end users, but given the scarcity of RGtk2 resources in the web, I thought they should be available in this package in the hope they could be usefull for other projects. Please feel free to use them under the terms of the package licence, but do not expect backward compatibility in future versions of this package. These functions are listed below: - askQuit - checkEntry - makeMainWindowsContent - makeMenuMainWindow - open_cb - open_cbFile - switchOnDialogWait - switchOffDialogWait ## 3. Going further From this point, considering words as species, word occurrences as abundances, and documents as communities, an incredible amount of analyses from theoretical ecology are available in R. Some examples are Rank-abundance curve, Species-Area Relationships, or Single large or Several small analyses. All of them provide interesting points to compare and analyse sets of documents.