web scraping research papers ieee

• Stack Overflow Public questions & answers
• Stack Overflow for Teams Where developers & technologists share private knowledge with coworkers
• Talent Build your employer brand
• Advertising Reach developers & technologists worldwide
• Labs The future of collective knowledge sharing
• About the company

Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Web scraping of research paper on IEEE Xplore website using BeautifulSoup and request Python libraries

I am trying to scrape the Abstract of the research paper on IEEE Xplore website, link : . For this I used urllib library and Beautifulsoup in Python(3.10.9). Below is the code i have used:

Here i have attached the screenshot of html part having Abstract.

I am getting AttributeError: 'NoneType' object has no attribute 'text' for abstract.

I got the value of soup. But I don't know how to get the Abstract text. I am new to web scraping. I have tried a lot but didn't able to solve it. Please help me to solve this problem. Thanks.

• beautifulsoup
• web-crawler

• when using requests , first check if you are really getting the html content you see on the browser. for example, if you write the received HTML to a file and then open that html file in the browser, you will notice the problem. write content to file with open("received_page.html","w") as fh: fh.write(str(soup)) , this is the reason that people use selenium for data extraction. –  simpleApp Jun 7 at 0:34

In this case it's actually possible to get the abstract without the need for Selenium. I generally use the requests library and css selectors in BeautifulSoup and that's what I did here:

and then simply:

The output should be the abstract.

• Thank you for your help. This worked for me. I am a beginner and still learning about web scraping. Could you please explain me about >'('meta[property="og:description"]')['content']')'. Like how to know that this has to be used? –  Devesh S Jun 7 at 4:40
• The above code doesn't give abstract in proper format for another [link] ( ieeexplore.ieee.org/document/9800203 ). Could you please once more help me. –  Devesh S Jun 7 at 5:00
• @DeveshS In the case of the 2nd document, the text of the abstract is, for some reason, sprinkled with non-well-formed xml tags, which don't render on the web page but do render on print . It's possible to extract the abstract from there by using a library such as lxml (although that probably should be presented in a separate question, per SO policy). But it's important that you realize that things like this are very likely to happen with web scraping. Page structures are changed frequently and sometimes radically, and you'll always have to play catch up with them. –  Jack Fleeting Jun 7 at 11:07
• Thanks for your response. I have cleaned the text with regex. –  Devesh S Jun 10 at 18:59
• @DeveshS As many people before me said, it's not a good idea to parse xml with regex.... . In any case, if we're done, please don't forget to accept the answer. –  Jack Fleeting Jun 10 at 22:45

Your Answer

Sign up or log in, post as a guest.

Required, but never shown

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct .

Not the answer you're looking for? Browse other questions tagged python beautifulsoup web-crawler urllib or ask your own question .

• The Overflow Blog
• Journey to the cloud part I: Migrating Stack Overflow Teams to Azure
• Featured on Meta
• Our Design Vision for Stack Overflow and the Stack Exchange network
• Temporary policy: Generative AI (e.g., ChatGPT) is banned
• Call for volunteer reviewers for an updated search experience: OverflowAI Search
• Discussions experiment launching on NLP Collective

Hot Network Questions

• How do languages support executing untrusted user code at runtime?
• Are there any good alternatives to firearms for 1920s aircrafts?
• Confused about the notion of overfitting and noisy target function
• Male and female seahorses: which is which
• What is the difference between computation and simulation?
• All arsenic not in the ground is magically removed. What effect would that have?
• Documents needed when travelling from the U.S. to Canada and back by land as a German citizen
• What だってそうだろう means here?
• Why is the convolution of two sine waves a sinc function?
• Why is there an indefinite article before the proper noun in "He lacked the analytic processing power of a Hamilton"?
• Sci-fi story involving telescopic technology that could reach out into space and view the universe from remote locations
• How to use Arabic voice?
• Is the area of the Mandelbrot set known?
• Does theism have the burden of proof?
• How does Legendre transformation in classical mechanics relate to Adrien-Marie Legendre?
• Why do STM32 MCUs divide RAM into SRAM1 and SRAM2?
• Coworker keeps telling me where to sit. How can I handle this?
• Meaning of "retiring" in "free admission with retiring donations"
• How to pass a statistics exam that overly relies on memorizing formulas?
• Risky Surgery and resistance to slashing damage
• File Organiser in Python
• Be big more often
• Why is the French embassy in Niger still considered an embassy?
• What does ggf reserviert mean on DB trains?

Your privacy

By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy .

Search code, repositories, users, issues, pull requests...

Provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

• Notifications

A web scraping tool to systematically extract the text of scientific papers and corresponding metadata from university accessible journals.

NLPatVCU/PaperScraper

Name already in use.

Use Git or checkout with SVN using the web URL.

Work fast with our official CLI. Learn more about the CLI .

• Open with GitHub Desktop
• Download ZIP

Sign In Required

Please sign in to use Codespaces.

Launching GitHub Desktop

If nothing happens, download GitHub Desktop and try again.

Launching Xcode

If nothing happens, download Xcode and try again.

Launching Visual Studio Code

Your codespace will open once ready.

There was a problem preparing your codespace, please try again.

Latest commit

PaperScraper

PaperScraper facilitates the extraction of text and meta-data from scientific journal articles for use in NLP systems. In simplest application, query by the URL of a journal article and receive back a structured JSON object containing the article text and metadata. More robustly, query by relevant attribute tags of articles (ie. DOI, Pubmed ID) and have an article URL automatically found and extracted from.

Retrieve structured journal articles in three lines:

or use a domain-specific aggregator such as PubMed and let PaperScraper automatically find a link for you:

Current Scraping Capabilities

Contribution.

To contribute an additional scraper to PaperScraper simply do the following (detailed instructions found in section 'Example Contribution Development Set-up'):

• Fork this repository and clone down a local version of your fork.
• set-up/enter a virtual environment using a Python version of 3.5 or greater.
• Run the setup.py file and verify that all package requirements have successfully installed in your virtual environment.
• Contribute a scraper by adding a file to the paperscraper/scrapers directory following the naming convention '<journal>_scraper.py'. Your scraper should implement the BaseScraper interface and simply include the necessary methods (see other scrapers for examples). The package will handle all other integration of your scraper into the framework.
• PaperScraper utilizes BeautifulSoup4 for navigating markdown. When writing a scraper, each method receives an instance of a BeautifulSoup object that wraps the markdown of the queried website. This markdown is then navigated to retrieve relevant information. See BeautifulSoup documentation and examples .
• While developing a scraper, one should simultaneously be write a test file under tests/scrapers named 'test_<journal>.py' . This will not only allow the concurrent debugging of each method as one develops, but more importantly allow for the identifications of any markdown changes (and resulting scraping errors) the publisher makes after development concludes. We emphasis, writing a test is vital to the longevity of your contribution and subsequently the package. See 'Detailed Contribution Instructions' for a walk-through of testing a contribution.
• Once complete, run all package tests with nosetests and submit a pull request.

Contribution Standards

Follow the following formatting standards when developing a scraper:

• Include all meta-html tags inside of the body such as links (<a>), emphasis <em>, etc. These can be filtered out by the end user but can also serve to provide meaningful information to some systems.
• The OrderedDict containing the paper body should be structured as follows:

Example Contribution Development Set-up

We recommend using an IDE such as PyCharm to facilitate the contribution process. It is available for free if you are affiliated with a university . This contribution walk-through assumes that you are utilizing PyCharm Professional Edition.

• Create a new PyCharm project named 'PaperScraper'. When selecting an interpreter, click the gear icon and create a new virtual environment (venv) in a version of Python greater than 3.5 ( details here ). A Python virtual environment serves to isolate your current development from all python packages and version already installed on your machine.
• Fork this repository, navigate to the directory of your project, and clone your fork into it.
• The PyCharm directory view should now update with all relevant project files. Press the button 'Terminal' in the lower-left corner of the IDE to open up an in-IDE terminal instance local to your project - notice the virtual environment is already set.
• Execute python setup.py to install PaperScraper and its dependencies into your virtual environment.
• Execute python setup.py test to run all tests. Insure that you have an internet connection as some tests require it. Further tests (along with only running single test files) can be executed by the command 'nosetests' ( details here ).
• Create new files '<journal>_scraper.py' and 'test_<journal>.py' in paperscraper/scrapers and tests/scrapers respectively. Model the structure/naming conventions of these files after other files in the directories.
• When testing your contribution-in-progress, run the command 'nosetesting -s <test_file_path>' to test only a single file. The '-s' parameter will allow print statements in your test files to show when tests are run. These should be removed before making a pull request.

Ensure that you have an internet connection before testing. To execute all tests, run the command python setup.py test from the top-level directory. To execute a single test, run the command nosetests -s <test_file_path> . The -s flag will allow print statements to print to console. Please remove all print statements before submitting a pull request.

Check out the Nose testing documentation here .

If you are experiencing errors running tests, make sure Nose is running with a version of python 3.5 or greater. If it is not, it is likely an error with Nose not being installed in your virtual environment. Execute the command pip install nose -I to correctly install it.

When writing tests, cover scraping from a few different correct and incorrect URLs. Also test that there is valid output for key sections such as 'authors' and 'body'. Please follow the naming convention for your test files . Refer to the test_sciencedirect.py file as a template for your own tests.

This package is licensed under the GNU General Public License

Acknowledgments

• Nanoinformatics Vertically Integrated Projects

Contributors 4

• Python 100.0%

Web Scraping & Automation Bot Using Python : Using Python to automate all the tasks

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2023 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.