• Stack Overflow Public questions & answers
  • Stack Overflow for Teams Where developers & technologists share private knowledge with coworkers
  • Talent Build your employer brand
  • Advertising Reach developers & technologists worldwide
  • Labs The future of collective knowledge sharing
  • About the company

Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Web scraping of research paper on IEEE Xplore website using BeautifulSoup and request Python libraries

I am trying to scrape the Abstract of the research paper on IEEE Xplore website, link : . For this I used urllib library and Beautifulsoup in Python(3.10.9). Below is the code i have used:

Here i have attached the screenshot of html part having Abstract.

I am getting AttributeError: 'NoneType' object has no attribute 'text' for abstract.

I got the value of soup. But I don't know how to get the Abstract text. I am new to web scraping. I have tried a lot but didn't able to solve it. Please help me to solve this problem. Thanks.

  • beautifulsoup
  • web-crawler

Devesh S's user avatar

  • when using requests , first check if you are really getting the html content you see on the browser. for example, if you write the received HTML to a file and then open that html file in the browser, you will notice the problem. write content to file with open("received_page.html","w") as fh: fh.write(str(soup)) , this is the reason that people use selenium for data extraction. –  simpleApp Jun 7 at 0:34

In this case it's actually possible to get the abstract without the need for Selenium. I generally use the requests library and css selectors in BeautifulSoup and that's what I did here:

and then simply:

The output should be the abstract.

Jack Fleeting's user avatar

  • Thank you for your help. This worked for me. I am a beginner and still learning about web scraping. Could you please explain me about >'('meta[property="og:description"]')['content']')'. Like how to know that this has to be used? –  Devesh S Jun 7 at 4:40
  • The above code doesn't give abstract in proper format for another [link] ( ieeexplore.ieee.org/document/9800203 ). Could you please once more help me. –  Devesh S Jun 7 at 5:00
  • @DeveshS In the case of the 2nd document, the text of the abstract is, for some reason, sprinkled with non-well-formed xml tags, which don't render on the web page but do render on print . It's possible to extract the abstract from there by using a library such as lxml (although that probably should be presented in a separate question, per SO policy). But it's important that you realize that things like this are very likely to happen with web scraping. Page structures are changed frequently and sometimes radically, and you'll always have to play catch up with them. –  Jack Fleeting Jun 7 at 11:07
  • Thanks for your response. I have cleaned the text with regex. –  Devesh S Jun 10 at 18:59
  • @DeveshS As many people before me said, it's not a good idea to parse xml with regex.... . In any case, if we're done, please don't forget to accept the answer. –  Jack Fleeting Jun 10 at 22:45

Your Answer

Sign up or log in, post as a guest.

Required, but never shown

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct .

Not the answer you're looking for? Browse other questions tagged python beautifulsoup web-crawler urllib or ask your own question .

  • The Overflow Blog
  • Journey to the cloud part I: Migrating Stack Overflow Teams to Azure
  • Featured on Meta
  • Our Design Vision for Stack Overflow and the Stack Exchange network
  • Temporary policy: Generative AI (e.g., ChatGPT) is banned
  • Call for volunteer reviewers for an updated search experience: OverflowAI Search
  • Discussions experiment launching on NLP Collective

Hot Network Questions

  • How do languages support executing untrusted user code at runtime?
  • Are there any good alternatives to firearms for 1920s aircrafts?
  • Confused about the notion of overfitting and noisy target function
  • Male and female seahorses: which is which
  • What is the difference between computation and simulation?
  • All arsenic not in the ground is magically removed. What effect would that have?
  • Documents needed when travelling from the U.S. to Canada and back by land as a German citizen
  • What だってそうだろう means here?
  • Why is the convolution of two sine waves a sinc function?
  • Why is there an indefinite article before the proper noun in "He lacked the analytic processing power of a Hamilton"?
  • Sci-fi story involving telescopic technology that could reach out into space and view the universe from remote locations
  • How to use Arabic voice?
  • Is the area of the Mandelbrot set known?
  • Does theism have the burden of proof?
  • How does Legendre transformation in classical mechanics relate to Adrien-Marie Legendre?
  • Why do STM32 MCUs divide RAM into SRAM1 and SRAM2?
  • Coworker keeps telling me where to sit. How can I handle this?
  • Meaning of "retiring" in "free admission with retiring donations"
  • How to pass a statistics exam that overly relies on memorizing formulas?
  • Risky Surgery and resistance to slashing damage
  • File Organiser in Python
  • Be big more often
  • Why is the French embassy in Niger still considered an embassy?
  • What does ggf reserviert mean on DB trains?

web scraping research papers ieee

Your privacy

By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy .

Search code, repositories, users, issues, pull requests...

Provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

  • Notifications

A web scraping tool to systematically extract the text of scientific papers and corresponding metadata from university accessible journals.


Name already in use.

Use Git or checkout with SVN using the web URL.

Work fast with our official CLI. Learn more about the CLI .

  • Open with GitHub Desktop
  • Download ZIP

Sign In Required

Please sign in to use Codespaces.

Launching GitHub Desktop

If nothing happens, download GitHub Desktop and try again.

Launching Xcode

If nothing happens, download Xcode and try again.

Launching Visual Studio Code

Your codespace will open once ready.

There was a problem preparing your codespace, please try again.

Latest commit



PaperScraper facilitates the extraction of text and meta-data from scientific journal articles for use in NLP systems. In simplest application, query by the URL of a journal article and receive back a structured JSON object containing the article text and metadata. More robustly, query by relevant attribute tags of articles (ie. DOI, Pubmed ID) and have an article URL automatically found and extracted from.

alt text

Retrieve structured journal articles in three lines:

or use a domain-specific aggregator such as PubMed and let PaperScraper automatically find a link for you:

Current Scraping Capabilities


To contribute an additional scraper to PaperScraper simply do the following (detailed instructions found in section 'Example Contribution Development Set-up'):

  • Fork this repository and clone down a local version of your fork.
  • set-up/enter a virtual environment using a Python version of 3.5 or greater.
  • Run the setup.py file and verify that all package requirements have successfully installed in your virtual environment.
  • Contribute a scraper by adding a file to the paperscraper/scrapers directory following the naming convention '<journal>_scraper.py'. Your scraper should implement the BaseScraper interface and simply include the necessary methods (see other scrapers for examples). The package will handle all other integration of your scraper into the framework.
  • PaperScraper utilizes BeautifulSoup4 for navigating markdown. When writing a scraper, each method receives an instance of a BeautifulSoup object that wraps the markdown of the queried website. This markdown is then navigated to retrieve relevant information. See BeautifulSoup documentation and examples .
  • While developing a scraper, one should simultaneously be write a test file under tests/scrapers named 'test_<journal>.py' . This will not only allow the concurrent debugging of each method as one develops, but more importantly allow for the identifications of any markdown changes (and resulting scraping errors) the publisher makes after development concludes. We emphasis, writing a test is vital to the longevity of your contribution and subsequently the package. See 'Detailed Contribution Instructions' for a walk-through of testing a contribution.
  • Once complete, run all package tests with nosetests and submit a pull request.

Contribution Standards

Follow the following formatting standards when developing a scraper:

  • Include all meta-html tags inside of the body such as links (<a>), emphasis <em>, etc. These can be filtered out by the end user but can also serve to provide meaningful information to some systems.
  • The OrderedDict containing the paper body should be structured as follows:

Example Contribution Development Set-up

We recommend using an IDE such as PyCharm to facilitate the contribution process. It is available for free if you are affiliated with a university . This contribution walk-through assumes that you are utilizing PyCharm Professional Edition.

  • Create a new PyCharm project named 'PaperScraper'. When selecting an interpreter, click the gear icon and create a new virtual environment (venv) in a version of Python greater than 3.5 ( details here ). A Python virtual environment serves to isolate your current development from all python packages and version already installed on your machine.
  • Fork this repository, navigate to the directory of your project, and clone your fork into it.
  • The PyCharm directory view should now update with all relevant project files. Press the button 'Terminal' in the lower-left corner of the IDE to open up an in-IDE terminal instance local to your project - notice the virtual environment is already set.
  • Execute python setup.py to install PaperScraper and its dependencies into your virtual environment.
  • Execute python setup.py test to run all tests. Insure that you have an internet connection as some tests require it. Further tests (along with only running single test files) can be executed by the command 'nosetests' ( details here ).
  • Create new files '<journal>_scraper.py' and 'test_<journal>.py' in paperscraper/scrapers and tests/scrapers respectively. Model the structure/naming conventions of these files after other files in the directories.
  • When testing your contribution-in-progress, run the command 'nosetesting -s <test_file_path>' to test only a single file. The '-s' parameter will allow print statements in your test files to show when tests are run. These should be removed before making a pull request.

Ensure that you have an internet connection before testing. To execute all tests, run the command python setup.py test from the top-level directory. To execute a single test, run the command nosetests -s <test_file_path> . The -s flag will allow print statements to print to console. Please remove all print statements before submitting a pull request.

Check out the Nose testing documentation here .

If you are experiencing errors running tests, make sure Nose is running with a version of python 3.5 or greater. If it is not, it is likely an error with Nose not being installed in your virtual environment. Execute the command pip install nose -I to correctly install it.

When writing tests, cover scraping from a few different correct and incorrect URLs. Also test that there is valid output for key sections such as 'authors' and 'body'. Please follow the naming convention for your test files . Refer to the test_sciencedirect.py file as a template for your own tests.

This package is licensed under the GNU General Public License


alt text

  • Nanoinformatics Vertically Integrated Projects

Contributors 4


  • Python 100.0%

Web Scraping & Automation Bot Using Python : Using Python to automate all the tasks

Ieee account.

  • Change Username/Password
  • Update Address

Purchase Details

  • Payment Options
  • Order History
  • View Purchased Documents

Profile Information

  • Communications Preferences
  • Profession and Education
  • Technical Interests
  • US & Canada: +1 800 678 4333
  • Worldwide: +1 732 981 0060
  • Contact & Support
  • About IEEE Xplore
  • Accessibility
  • Terms of Use
  • Nondiscrimination Policy
  • Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2023 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.


  1. Ieee Research Papers Website

    web scraping research papers ieee

  2. Iot Paper Ieee Format

    web scraping research papers ieee

  3. (PDF) Data Analysis by Web Scraping using Python

    web scraping research papers ieee

  4. IEEE format sample

    web scraping research papers ieee

  5. Ieee Paper Review Format

    web scraping research papers ieee

  6. How To Download IEEE Papers From IEEEXPLORE Website For Free

    web scraping research papers ieee


  1. Learn web scraping and earn

  2. Add a Database to your Web Scraper

  3. Web Scrapping on College Website for News using Python

  4. Web Scraping ScienceDirect.com Articles

  5. Pagination in Web Scraping

  6. What is WEB SCRAPING and How it Works #webscraping #technology #trending #pixipace


  1. A Review on Web Scrapping and its Applications

    A Review on Web Scrapping and its Applications Abstract: Internet grants a wide scope of facts and data source established by humans. Though, it shall consist of an enormous assortment of dissimilar and ailing organized data, challenging in collection in a physical means and problematical for its usage in mechanical processes.

  2. Web Scraping Approaches and their Performance on Modern Websites

    This paper contains research-based findings of different methods of web scraping techniques used to extract data from websites. The various approaches used in this paper to obtain the results include requests library, selenium and other external libraries.

  3. Web Scraping: State-of-the-Art and Areas of Application

    Abstract: Main objective of Web Scraping is to extract information from one or many websites and process it into simple structures such as spreadsheets, database or CSV file. However, in addition to be a very complicated task, Web Scraping is resource and time consuming, mainly when it is carried out manually.

  4. Data Analysis by Web Scraping using Python

    Data Analysis by Web Scraping using Python Abstract: The standard information investigation are built on the root and impact relationship, shaped an example minuscule examination, subjective and quantitative examination, the rationality approach of creating extrapolation examination.

  5. Web Scraping And Data Acquisition Using Google Scholar

    This paper attempts to set up an interface that would use web scraping techniques and Python modules to link a researcher's list of publications present on Google Scholar to a MySQL database and Excel application, allowing them to access and manipulate their works in minimal steps.

  6. Web Scraping for Data Analytics: A BeautifulSoup Implementation

    Web scraping is an essential tool for automating the data-gathering process for big data applications. There are many implementations for web scraping, but barely any of them is based on Python's BeautifulSoup library. Therefore, this paper aims at creating a web scraper that gathers data from any website and then analyzes the data accordingly. For results and analysis, the web scraper has ...

  7. Web scrapping and storing data in a database, a case ...

    In this paper we will deal with the process of web scraping data from different locations on the Internet and their storage in a database, for the purpose of collecting and analyzing data of the used cars market. Published in: 2017 25th Telecommunication Forum (TELFOR) Article #: Date of Conference: 21-22 November 2017

  8. A Web Scraping based approach for data research through ...

    This paper presents a Web Scraping based approach to solve the data extraction problem in Instagram. Data from Instagram can be helpful in different research contexts. We describe the theoretical characterization of our proposal and its implementation in a real-case scenario. Also, we provide preliminary results and potential implementation challenges. The main contribution of our work is an ...

  9. Web Scraping Using Summarization and Named Entity ...

    In this paper Natural Language Processing (NLP) and Machine Learning (ML) alternatives to the traditional web-scraping approach is presented. To demonstrate the advantages offered by the improved algorithm, an epidemic predictor mapping the spread of a variety of infectious/viral diseases and their impact across the globe is built using the ...

  10. Utilizing Web Scraping and Natural Language Processing to ...

    In this work, we apply connectivism as the theoretical framework, and demonstrate how web scraping can be useful for extrapolating large amounts of data from publicly available web pages to pool data from a wider array of sources and to further knowledge in the field.

  11. An Optimal Data Entry Method, Using Web Scraping and Text ...

    In this paper, we propose a text recognition system that can be employed to detect text from images automatically and update it to a target file. The proposed method accepts a web URL as the input and fetches the text or image using web scraping technique. The system extracts textual data from a user specified region.

  12. A dive into Web Scraper world

    This paper talks about the World of Web Scraper, Web scraping is related to web indexing, whose task is to index information on the web with the help of a bot or web crawler. Here the legal aspect, both positive and negative sides are taken into view. Some cases regarding the legal issues are also taken into account. The Web Scraper's designing principles and methods are contrasted, it tells ...

  13. (PDF) Web Data Scraping

    Web scraping, or web scratching, is a procedure which is utilized to create organized information based on accessible unstructured information on the web. Created organized information at...

  14. Full article: Web Scraping in the Statistics and Data Science

    Teaching web scraping provides an opportunity to bring such data into the curriculum in an effective and efficient way. In this article, we explain how web scraping works and how it can be implemented in a pedagogically sound and technically executable way at various levels of statistics and data science curricula.

  15. Utilizing Web Scraping and Natural Language Processing to Better Inform

    2020 IEEE Frontiers in Education Conference (FIE) ... This research full paper describes how web scraping and natural language processing can be utilized to answer complex questions in computer science education. In this work, we apply connectivism as the theoretical framework, and demonstrate how web scraping can be useful for extrapolating ...

  16. web scraping

    1 Answer Sorted by: 2 When you are clicking on the search button, it is redirecting you to a link https://ieeexplore.ieee.org/search/searchresult.jsp?newsearch=true&queryText=smart%20grid The queryText parameter is your search term. The content is loaded using JavaScript, so you cannot just send a request to the link and then parse the response.

  17. Web scraping of research paper on IEEE Xplore website using

    1 Answer Sorted by: 0 In this case it's actually possible to get the abstract without the need for Selenium. I generally use the requests library and css selectors in BeautifulSoup and that's what I did here: import requests req = requests.get (url, headers=headers) soup = BeautifulSoup (req.text, "html.parser")

  18. GitHub

    GitHub - NLPatVCU/PaperScraper: A web scraping tool to systematically extract the text of scientific papers and corresponding metadata from university accessible journals. PaperScraper master 3 branches 0 tags Code Dendendelen Forgot to import it, but it's there now! 47399fb on Dec 22, 2020 46 commits docs 5 years ago 5 years ago .gitignore

  19. (PDF) A Novel Web Scraping Approach Using the Additional Information

    Web scraping is a process of extracting valuable and interesting text information from web pages. Most of the current studies targeting this task are mostly about automated web data...

  20. Web Scraping based Product Comparison Model for E ...

    Web Scraping based Product Comparison Model for E-Commerce Websites. 10.1109/ICDSIS55133.2022.9915892. Conference: 2022 IEEE International Conference on Data Science and Information System (ICDSIS)

  21. PDF Web Scraping: Applications and Scraping Tools

    ABSTRACT The world of Artificial Intelligence and machine learning has its common roots with data, which is primarily the most important entity on its own. Data has already impacted so many businesses worldwide and can never take a back seat when it comes to this technical world.

  22. PDF ISSN : 2454-9150 Data Analysis by Web Scraping using Python

    The scraper tool for the web is utilized for derived information from the web host, and as a portion of uses used for web orders, web mining and data mining, online esteem change observing and value correlation, element survey scratching (to watch the challenge),gathering land postings, atmosphere data checking, webpage change area, inspect, fol...

  23. Web Scraping & Automation Bot Using Python

    This research Paper is the work of our Automation Framework using Python which aims to provide a completely automated solution by automating each and every functionality and how the concept of automation has changed the traditional software working and processing. As technology usage is increasing, web applications can be automated to provide efficient services to more customers that saves ...