The automatic identification of stop words
Abstract
A stop word may be identified as a word that has the same likehhood of occurring in those documents not relevant to a query as in those documents relevant to the query. In this paper we show how the concept of relevance may be replaced by the condition of being highly rated by a similarity measure. Thus it becomes possible to identify the stop words in a cullectmn by automated statistical testing. We describe the nature of the statistical test as it is realized with a vector retrieval methodology based on the cosine coefficient of document-document similarity. As an example, this tech nique is then applied to a large MEDLINE " subset in the area of biotechnology. The initial processing of this datahase involves a 310 word stop list of common non-content terms. Our technique is then applied and 75% of the remaining terms are identified as stop words. We compare retrieval with and without the removal of these stop words and find that of the top twenty documents retrieved in response to a random query document. seventeen of these are the same on the average for the two methods We also examine the differences and conclude that where the user prefers one method over the other, the new method with the reduced term set is favored about three times out of four.
Get full access to this article
View all access and purchase options for this article.
References
C. Buckley, Implementation of the SMART information retrieval system, Technical Report 85-686 (Department of Computer Science, Cornell University, 1985).
C. Buckley and A.F. Lewit, Optimization of inverted vector searches. In. Proceedings of the Eighth International ACM Conference on Research and Development in Information Retrietal (Montreal. Quebec, 1985) 97-110.
W.B. Croft, Experiments with representation in a document retrieval system, Technical Report 82-21 (COINS. University of Massachusetts, Amherst, MA, 1982).
D. Lucarella, A document retrieval system based on nearest neighbor searching, Journal of Information Science 14 (1988) 25-33.
M.F. Porter, An algorithm tor suffiy stripping, Program 14 (1980) 130-137.
J.J. Regazzi, Performance measures for information retrieval systems: an experimental approach, Journal of the American Suciety for Information Science 39(4) (1988) 235-251.
C.J. van Rijsbergen, Information Retriet al. 2nd ed (Butteworths, London, 1979).
G. Salton. Automatic Information Organization and Retrietal (McGraw-Hill. New York, 1968).
G. Salton, A. Wong and C.S. Yang. A vector space model for automatic indexing, Communications of the ACM 18(11) (1975) 613-620.
G. Salton and M. McGill, Introduction to Modern Information Retrieval (McGraw-Hill, New York, 1983).
G. Salton, Automatic Text Processing (Addison-Wesley. Reading, MA, 1989).
K Sparck Jones, A statistical interpretation of term specificity and its application, Journal of Documentation 28 (1972) 11-21.
D.R. Swanson, Historical note information retrieval and the future of an illusion, Journal of the American Society for Information Science 39(2) (1988) 92-98.
S.K.M. Wong and V.V. Raghaven. Vector space model of information retrieval: a reevaluation In: C.J. van Rijsbergen, ed. Research and Development in Information Retrieval (Cambridge University Press. Cambridge, 1984 ) 167-185.
Cite article
Cite article
Cite article
OR
Download to reference manager
If you have citation software installed, you can download article citation data to the citation manager of your choice
Information, rights and permissions
Information
Published In
Article first published: February 1992
Issue published: February 1992
History
Published online: February 1, 1992
Issue published: February 1992
Authors
Metrics and citations
Metrics
Journals metrics
This article was published in Journal of Information Science.
VIEW ALL JOURNAL METRICSArticle usage*
Total views and downloads: 463
*Article usage tracking started in December 2016
Altmetric
See the impact this article is making through the number of times it’s been read, and the Altmetric Score.
Learn more about the Altmetric Scores
Articles citing this one
Web of Science: 139 view articles Opens in new tab
Crossref: 153
- A dataset on corporate sustainability disclosure
- Multi-class sentiment classification on Bengali social media comments ...
- Calculation of embodied GHG emissions in early building design stages ...
- Sentiment analysis of medical record notes for lung cancer patients at...
- Network-Based Dimensionality Reduction for Textual Datasets
- Sigmoidal Particle Swarm Optimization for Twitter Sentiment Analysis
- A Proposed Method of Literature Analysis Based on Natural Language Pro...
- Automatic keyword extraction for localized tweets using fuzzy graph co...
- Using neutral sentiment reviews to improve customer requirement identi...
- Prompt engineering for zero‐shot and few‐shot defect detection and cla...
- Using Topic Models to Understand Rater-Mediated Writing Assessments
- What is an independent art space? Using a text-mining approach to desc...
- Pricing the Long Tail by Explainable Product Aggregation and Monotonic...
- BIM-based design decisions documentation using design episodes, explan...
- Natural language processing in low back pain and spine diseases: A sys...
- Automatic identification of sentiment in unstructured text
- Citizens at the forefront of the constitutional debate: Voluntary citi...
- Classification of open-ended responses to a research-based assessment ...
- Research on Passengers’ Preference for High-Speed Railways (HSRs) and ...
- A Practical Tutorial for Decision Tree Induction
- A Proposed Bi-LSTM Method to Fake News Detection
- Exploring Rater Accuracy Using Unfolding Models Combined with Topic Mo...
- One-Word Approach in Text-Mining for Value Identification
- Text Clustering
- Police narrative reports: Do they provide end-users with the data they...
- What Can Social Media Tell Us About Patient Symptoms
- Semantic and Sentiment Analysis of Selected Bhagavad Gita Translations...
- A Novel Dictionary Generation Methodology for Contextual-Based Passwor...
- MathSBERT: A Language Representation Model for Mathematical Informatio...
- Stop words detection using a long short term memory recurrent neural n...
- Occupants’ satisfaction with LEED- and non-LEED-certified apartments u...
- AuTGeLy: Automatic Title Generator based on Song Lyrics Extractions
- Machine learning in medicine: a practical introduction to natural lang...
- Entropic measures of complexity in a new medical coding system
- Mapping the genealogy of medical device predicates in the United State...
- BenSW: A Standard Dataset for Bengali Stop Word Detection
- Twitter sentiment analysis using hybrid Spider Monkey optimization met...
- Automatic Multilingual Stopwords Identification from Very Small Corpor...
- Stopwords in technical language processing
- Comparative Analysis of Bengali Stop Word Detection Using Different Ap...
- DYNAMIC STOP LIST FOR THE GUJARATI LANGUAGE USING RULE BASED APPROACH
- Automatic Stopwords Identification from Very Small Corpora
- Bengali Stop Word Detection Using Different Machine Learning Algorithm...
- dh2loop 1.0: an open-source Python library for automated processing an...
- Creating a stop word dictionary in Serbian
- Synthetic minority oversampling in addressing imbalanced sarcasm detec...
- Assisted authoring of model-based systems engineering documents
- Automatic offensive language detection from Twitter data using machine...
- The Art of Feature Engineering
- Microfeatures influencing writing quality: the case of Chinese student...
- The Challenges of Designing a Robot for a Satisfaction Survey: Surveyi...
- Convolutional neural network model based on text similarity for custom...
- A Novel Short Text Clustering Model Based on Grey System Theory
- Organizational context and budget orientations: a computational text a...
- Essential Elements of Natural Language Processing: What the Radiologis...
- An Efficient Topic Modeling Approach for Text Mining and Information R...
- Automatic Stopword Detection Using Term Ranking between Written and Ma...
- Online health community experiences of sexual minority women with canc...
- Temporal topic modeling applied to aviation safety reports: A subject ...
- H-Rank: A keywords extraction method from web pages using POS tags
- S3BD: Secure semantic search over encrypted big data in the cloud
- Big Social Data - Predicting Users' Interests from their Social Networ...
- LENN: Laplacian Probability Based Extended Nearest Neighbor Classifica...
- Sentiment Classification of Customer’s Reviews About Automobiles in Ro...
- Text Preprocessing
- Arabic Web page clustering: A review
- A Study on Effective Measurement of Search Results from Search Engines
- Pairwise document similarity measure based on present term set
- Discovering IMRaD Structure with Different Classifiers
- Estimating Similarity Among Entities Aided by the Web when Only the En...
- Multi-Label Classification of Contributing Causal Factors in Self-Repo...
- Dictionaries and distributions: Combining expert knowledge and large s...
- Machine Learning Implementations in Arabic Text Classification
- On Frequency-Based Approaches to Learning Stopwords and the Reliabilit...
- Text Clustering
- Risk Assessment for Parents Who Suspect Their Child Has Autism Spectru...
- A Brief Study of Approaches to Text Feature Selection
- Using Twitter and the mobile cloud for delivering medical help in emer...
- A feature selection method based on synonym merging in text classifica...
- Automatic classification of journalistic documents on the Internet1
- The aboutness of words
- Comparing grounded theory and topic modeling: Extreme divergence or un...
- Programming Tools for Messenger-Based Chatbot System Organization: Imp...
- Exploring Online Ad Images Using a Deep Convolutional Neural Network A...
- Computational Text Analysis for Public Management Research
- Fostering parent–child dialog through automated discussion suggestions
- Leveraging Topic Model for CSI Based Human Activity Recognition
- Landmark Reranking for Smart Travel Guide Systems by Combining and Ana...
- Conceptualizing Big Data: Analysis of Case Studies
- EXAF: A search engine for sample applications of object-oriented frame...
- A New Feature Selection Approach to Naive Bayes Text Classifiers
- A Method for Measuring Similarity of Books: A Step Towards an Objectiv...
- Design and Use of a Semantic Similarity Measure for Interoperability A...
- Interactive Big Data Visualization Model Based on Hot Issues (Online N...
- Core Informatics Technologies: Data Storage
- Text mining: An improvised feature based model approach
- Using Social Media and the Mobile Cloud to Enhance Emergency and Risk ...
- Introducing Connected Concept Analysis: A network approach to big text...
- Supervised machine learning for the detection of troll profiles in twi...
- Using compression models for filtering troll comments
- Visual Analysis of Topical Evolution in Unstructured Text: Design and ...
- Author Topic Model based Collaborative Filtering for Personalized POI ...
- An Information Theoretic Clustering Approach for Unveiling Authorship ...
- Improving NCD accuracy by combining document segmentation and document...
- Study on the effectiveness of anomaly detection for spam filtering
- Crowdsourced weather reports: An implementation of the μ model ...
- An unsupervised cascade learning scheme for ‘cluster-theme keywords’ s...
- Text-based emotion classification using emotion cause extraction
- A Heuristic Attribute Reduction Based on Multi-Granularity Rough Set
- Supervised Machine Learning for the Detection of Troll Profiles in Twi...
- Improving Near-Duplicate Detection in Multi-Layered Collaborative Requ...
- An Ant Colony Optimization Based Feature Selection for Web Page Classi...
- An Intelligent Content Discovery Technique for Health Portal Content M...
- A comparison of different calculations for N-gram similarities in a sp...
- Collective classification for spam filtering
- A Practical Approach for Content Mining of Tweets
- Multiphase text mining predictor for market analysis
- Language Individuation and Marker Words: Shakespeare and His Maxwell's...
- A health information recommender system: Enriching YouTube health vide...
- COMBINATION OF MULTIPLE FEATURE SELECTION METHODS FOR TEXT CATEGORIZAT...
- Adult Content Filtering through Compression-Based Text Classification
- JURD: Joiner of Un-Readable Documents to reverse tokenization attacks ...
- Automatic categorisation of comments in social news websites
- Towards a more efficient and personalised advertisement content in on-...
- Is the contextual information relevant in text clustering by compressi...
- Word sense disambiguation for spam filtering
- A Survey of Text Clustering Algorithms
- Spam Filtering through Anomaly Detection
- Enhanced Topic-based Vector Space Model for semantics-aware spam filte...
- On the study of anomaly-based spam filtering using spam as representat...
- Low-Power Themes Classifier (LPTC): A Human-Expert-Based Approach for ...
- A model to identify mathematics topics in MXit lingo to provide tutors...
- Enhancing scalability in anomaly-based email spam filtering
- A new partitioning based algorithm for document clustering
- Reducing the Loss of Information through Annealing Text Distortion
- Text stream clustering algorithm based on adaptive feature selection
- Collective Classification for Spam Filtering
- Finding related sentence pairs in MEDLINE
- Beyond Redundancies: A Metric-Invariant Method for Unsupervised Featur...
- A delimiter-based general approach for Chinese term extraction
- Integrating Information Extraction Agents into a Tourism Recommender S...
- Relevance of Contextual Information in Compression-Based Text Clusteri...
- Combining Multiple Feature Selection Methods for Text Categorization b...
- Divergence-based feature selection for naïve Bayes text classif...
- Text Clustering with Feature Selection by Using Statistical Data
- Clustering methodologies for identifying country core competencies
- Using Links to Aid Web Classification
- Incorporating context in text analysis by interactive activation with ...
- Factor matrix text filtering and clustering
- Corpus-based statistical screening for content-bearing terms
- An analysis of statistical term strength and its use in the indexing a...
- An information measure of retrieval performance
- Generating titles for paragraphs using statistically extracted keyword...
Figures and tables
Figures & Media
Tables
View Options
Get access
Access options
If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:
loading institutional access options
CILIP members can access this journal content using society membership credentials.
CILIP members can access this journal content using society membership credentials.
Alternatively, view purchase options below:
Purchase 24 hour online access to view and download content.
Access journal content via a DeepDyve subscription or find out more about this option.
