+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Twitter scraping exclude some data (/thread-4599.html)
Twitter scraping exclude some data - Robbert - Aug-29-2017
Hello everyone! Im new here and im also new to python. Im eager to learn python because the possibilities are immense. Currently im working on a twitter streaming code, which I pasted in the code section below.
Im wondering how I should exclude data from the streamer?
1. For instance, i want to check wether the 'status' or 'location' fields are not null.
2. I would like to exclude some fields. For instance, 'retweets'.
If someone could explain how I'm supposed to program [1] en [2] then I would be very happy :)
from tweepy import Stream
from tweepy import OAuthHandler
from tweepy.streaming import StreamListener
import json
# consumer key, consumer secret, access token, access secret.
consumer_key = "xxx"
consumer_secret = "xxx"
access_token = "xxxx"
access_token_secret = "xxxx"
class StdOutlistener(StreamListener):
def on_data(self, data):
json_data = json.loads(data)
print (json_data)
# Open json text file to save the tweets
with open('tweets.json', 'a') as tf:
tf.write(data)
return True
def on_error(self, status):
print(status)
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
twitterStream = Stream(auth, StdOutlistener())
twitterStream.filter(track=["Test"])
RE: Twitter scraping exclude some data - Robbert - Aug-31-2017
Can anyone help me?
Is my question unclear>?
RE: Twitter scraping exclude some data - nilamo - Aug-31-2017
Are those fields part of the json response you're receiving?
RE: Twitter scraping exclude some data - Robbert - Aug-31-2017
(Aug-31-2017, 05:33 PM)nilamo Wrote: Are those fields part of the json response you're receiving?
Yes. The json format contains all the data that is available.
In this tutorial: http://adilmoujahid.com/posts/2014/07/twitter-analytics/ an overview is given of the data and Json output when no filters are applied. Literally everything is passing through and I would like to know whether it is possible to skip some fields. For instance; im not interesse in the fact that someone does have 20 followers or something.
RE: Twitter scraping exclude some data - nilamo - Aug-31-2017
Here's a direct link to the StreamListener class from the tweepy module: https://github.com/tweepy/tweepy/blob/v3.5.0/tweepy/streaming.py#L30
You're currently using on_data(), which fires off for every single message. Have you tried using one of the more specific ones, like on_status()?
RE: Twitter scraping exclude some data - Robbert - Sep-02-2017
(Aug-31-2017, 08:55 PM)nilamo Wrote: Here's a direct link to the StreamListener class from the tweepy module: https://github.com/tweepy/tweepy/blob/v3.5.0/tweepy/streaming.py#L30
You're currently using on_data(), which fires off for every single message. Have you tried using one of the more specific ones, like on_status()?
Thanks for your reply and suggestion. I will have at the webpage you mentioned.
No, i haven't tried on_status which would probably be better. But i have no idea how to use on_status in this particular script.
Do you perhaps have a link for that to?
RE: Twitter scraping exclude some data - nilamo - Sep-02-2017
You currently use on_data. replace the word "data" with "status", and it should run.