Proxy Server - Modules can't connect to URL - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: Proxy Server - Modules can't connect to URL (/thread-963.html) |
Proxy Server - Modules can't connect to URL - robmog - Nov-16-2016 Hi All I have setup Python3 in a virtualenv on Centos7. I'm at work and behind an HTTP proxy server. I have been trying to set up CVE-SEARCH, and I'm following instructions from github: (I'm not allowed links yet but its https:) //github.com/cve-search/cve-search/blob/master/README.md I've got to the section on populating the database and when I try the first command I get a Traceback shown below. I am connected to the internet and I guess it is failing because it needs to know to use the proxy - I could be wrong (: Appreciate if anyone could point me in the right direction. Kind regards Rob (my_env) [rmog@localhost environments]$ ./cve-search/sbin/db_mgmt.py -p Database population started During handling of the above exception, another exception occurred: (my_env) [rmog@localhost environments]$ idle ./cve-search/sbin/db_mgmt.py
RE: Proxy Server - Modules can't connect to URL - JChris - Nov-16-2016 Well, it seems your problem lies in the 'Configuration.getCVEDict()' call. You should take a look inside your class to see if the method 'getCVEDict()' is propersly defined. What is 'Configuration', is it a name assigned to an object of some class? RE: Proxy Server - Modules can't connect to URL - robmog - Nov-16-2016 (Nov-16-2016, 02:59 PM)JChris Wrote: Well, it seems your problem lies in the 'Configuration.getCVEDict()' call. You should take a look inside your class to see if the method 'getCVEDict()' is propersly defined. What is 'Configuration', is it a name assigned to an object of some class? Hi JChris Thanks for replying. Should have added in original post that I'm a newbie. From what I can see Configuration is imported from lib.Config. Should I be able to look into lib.Config? from lib.Config import ConfigurationThanks again Rob I have found the lib.Config file - and I see Configuration() but I've searched and I don't see getCVEDict anywhere -Am I looking in the wrong place? (my_env) [rmogan@localhost environments]$ ls -a cve-search/lib . Config.py PluginManager.py Toolkit.py .. cpelist.py Plugins.py User.py authenticationMethods CVEs.py ProgressBar.py Authentication.py DatabaseLayer.py Query.py (my_env) [rmogan@localhost environments]$ idle cve-search/lib/Config.py #!/usr/bin/env python3 # -*- coding: utf-8 -*- # # Config reader to read the configuration file # # Software is free software released under the "Modified BSD license" # # Copyright (c) 2013-2014 Alexandre Dulaunoy - [email protected] # Copyright (c) 2014-2016 Pieter-Jan Moreels - [email protected] # imports import sys import os runPath = os.path.dirname(os.path.realpath(__file__)) import pymongo import redis import bz2 import configparser import datetime import gzip import re import urllib.parse import urllib.request as req import zipfile from io import BytesIO class Configuration(): ConfigParser = configparser.ConfigParser() ConfigParser.read(os.path.join(runPath, "../etc/configuration.ini")) default = {'redisHost': 'localhost', 'redisPort': 6379, 'redisVendorDB': 10, 'redisNotificationsDB': 11, 'redisRefDB': 12, 'mongoHost': 'localhost', 'mongoPort': 27017, 'mongoDB': "cvedb", 'mongoUsername': '', 'mongoPassword': '', 'flaskHost': "127.0.0.1", 'flaskPort': 5000, 'flaskDebug': True, 'pageLength': 50, 'loginRequired': False, 'listLogin': True, 'ssl': False, 'sslCertificate': "./ssl/cve-search.crt", 'sslKey': "./ssl/cve-search.crt", 'CVEStartYear': 2002, 'logging': True, 'logfile': "./log/cve-search.log", 'maxLogSize': '100MB', 'backlog': 5, 'Indexdir': './indexdir', 'updatelogfile': './log/update.log', 'Tmpdir': './tmp', 'http_proxy': '', 'plugin_load': './etc/plugins.txt', 'plugin_config': './etc/plugins.ini', 'auth_load': './etc/auth.txt' } sources={'cve': "https://static.nvd.nist.gov/feeds/xml/cve/", 'cpe': "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.2.xml", 'cwe': "http://cwe.mitre.org/data/xml/cwec_v2.8.xml.zip", 'd2sec': "http://www.d2sec.com/exploits/elliot.xml", 'vendor': "https://nvd.nist.gov/download/vendorstatements.xml.gz", 'capec': "http://capec.mitre.org/data/xml/capec_v2.6.xml", 'msbulletin': "http://download.microsoft.com/download/6/7/3/673E4349-1CA5-40B9-8879-095C72D5B49D/BulletinSearch.xlsx", 'ref': "https://cve.mitre.org/data/refs/refmap/allrefmaps.zip", 'exploitdb': "https://github.com/offensive-security/exploit-database/raw/master/files.csv", 'includecve': True, 'includecapec': True, 'includemsbulletin': True, 'includecpe': True, 'included2sec': True, 'includeref': True, 'includecwe': True, 'includevendor': True, 'includeexploitdb': True} @classmethod def readSetting(cls, section, item, default): result = default try: if type(default) == bool: result = cls.ConfigParser.getboolean(section, item) elif type(default) == int: result = cls.ConfigParser.getint(section, item) else: result = cls.ConfigParser.get(section, item) except: pass return result # Mongo @classmethod def getMongoDB(cls): return cls.readSetting("Mongo", "DB", cls.default['mongoDB']) @classmethod def getMongoConnection(cls): mongoHost = cls.readSetting("Mongo", "Host", cls.default['mongoHost']) mongoPort = cls.readSetting("Mongo", "Port", cls.default['mongoPort']) mongoDB = cls.getMongoDB() mongoUsername = cls.readSetting("Mongo", "Username", cls.default['mongoUsername']) mongoPassword = cls.readSetting("Mongo", "Password", cls.default['mongoPassword']) mongoUsername = urllib.parse.quote( mongoUsername ) mongoPassword = urllib.parse.quote( mongoPassword ) try: if mongoUsername and mongoPassword: mongoURI = "mongodb://{username}:{password}@{host}:{port}/{db}".format( username = mongoUsername, password = mongoPassword, host = mongoHost, port = mongoPort, db = mongoDB ) connect = pymongo.MongoClient(mongoURI) else: connect = pymongo.MongoClient(mongoHost, mongoPort) except: sys.exit("Unable to connect to Mongo. Is it running on %s:%s?"%(mongoHost,mongoPort)) return connect[mongoDB] @classmethod def toPath(cls, path): return path if os.path.isabs(path) else os.path.join(runPath, "..", path) # Redis @classmethod def getRedisHost(cls): return cls.readSetting("Redis", "Host", cls.default['redisHost']) @classmethod def getRedisPort(cls): return cls.readSetting("Redis", "Port", cls.default['redisPort']) @classmethod def getRedisVendorConnection(cls): redisHost = cls.getRedisHost() redisPort = cls.getRedisPort() redisDB = cls.readSetting("Redis", "VendorsDB", cls.default['redisVendorDB']) return redis.StrictRedis(host=redisHost, port=redisPort, db=redisDB, charset='utf-8', decode_responses=True) @classmethod def getRedisNotificationsConnection(cls): redisHost = cls.getRedisHost() redisPort = cls.getRedisPort() redisDB = cls.readSetting("Redis", "NotificationsDB", cls.default['redisNotificationsDB']) return redis.StrictRedis(host=redisHost, port=redisPort, db=redisDB, charset="utf-8", decode_responses=True) @classmethod def getRedisRefConnection(cls): redisHost = cls.getRedisHost() redisPort = cls.getRedisPort() redisDB = cls.readSetting("Redis", "RefDB", cls.default['redisRefDB']) return redis.StrictRedis(host=redisHost, port=redisPort, db=redisDB, charset="utf-8", decode_responses=True) # Flask @classmethod def getFlaskHost(cls): return cls.readSetting("Webserver", "Host", cls.default['flaskHost']) @classmethod def getFlaskPort(cls): return cls.readSetting("Webserver", "Port", cls.default['flaskPort']) @classmethod def getFlaskDebug(cls): return cls.readSetting("Webserver", "Debug", cls.default['flaskDebug']) # Webserver @classmethod def getPageLength(cls): return cls.readSetting("Webserver", "PageLength", cls.default['pageLength']) # Authentication @classmethod def loginRequired(cls): return cls.readSetting("Webserver", "LoginRequired", cls.default['loginRequired']) @classmethod def listLoginRequired(cls): return cls.readSetting("Webserver", "ListLoginRequired", cls.default['listLogin']) @classmethod def getAuthLoadSettings(cls): return cls.toPath(cls.readSetting("Webserver", "authSettings", cls.default['auth_load'])) # SSL @classmethod def useSSL(cls): return cls.readSetting("Webserver", "SSL", cls.default['ssl']) @classmethod def getSSLCert(cls): return cls.toPath(cls.readSetting("Webserver", "Certificate", cls.default['sslCertificate'])) @classmethod def getSSLKey(cls): return cls.toPath(cls.readSetting("Webserver", "Key", cls.default['sslKey'])) # CVE @classmethod def getCVEStartYear(cls): date = datetime.datetime.now() year = date.year + 1 score = cls.readSetting("CVE", "StartYear", cls.default['CVEStartYear']) if score < 2002 or score > year: print('The year %i is not a valid year.\ndefault year %i will be used.' % (score, cls.default['CVEStartYear'])) score = cls.default['CVEStartYear'] return cls.readSetting("CVE", "StartYear", cls.default['CVEStartYear']) # Logging @classmethod def getLogfile(cls): return cls.toPath(cls.readSetting("Logging", "Logfile", cls.default['logfile'])) @classmethod def getUpdateLogFile(cls): return cls.toPath(cls.readSetting("Logging", "Updatelogfile", cls.default['updatelogfile'])) @classmethod def getLogging(cls): return cls.readSetting("Logging", "Logging", cls.default['logging']) @classmethod def getMaxLogSize(cls): size = cls.readSetting("Logging", "MaxSize", cls.default['maxLogSize']) split = re.findall('\d+|\D+', size) try: if len(split) > 2 or len(split) == 0: raise Exception base = int(split[0]) if len(split) == 1: multiplier = 1 else: multiplier = (split[1]).strip().lower() if multiplier == "b": multiplier = 1 elif multiplier == "kb": multiplier = 1024 elif multiplier == "mb": multiplier = 1024 * 1024 elif multiplier == "gb": multiplier = 1024 * 1024 * 1024 else: # If we cannot interpret the multiplier, we take MB as default multiplier = 1024 * 1024 return base * multiplier except Exception as e: print(e) return 100 * 1024 @classmethod def getBacklog(cls): return cls.readSetting("Logging", "Backlog", cls.default['backlog']) # Indexing @classmethod def getTmpdir(cls): return cls.toPath(cls.readSetting("dbmgt", "Tmpdir", cls.default['Tmpdir'])) # Indexing @classmethod def getIndexdir(cls): return cls.toPath(cls.readSetting("FulltextIndex", "Indexdir", cls.default['Indexdir'])) # Http Proxy @classmethod def getProxy(cls): return cls.readSetting("Proxy", "http", cls.default['http_proxy']) @classmethod def getFile(cls, getfile): if cls.getProxy(): proxy = req.ProxyHandler({'http': cls.getProxy(), 'https': cls.getProxy()}) auth = req.HTTPBasicAuthHandler() opener = req.build_opener(proxy, auth, req.HTTPHandler) req.install_opener(opener) response = req.urlopen(getfile) data = response if 'gzip' in response.info().get('Content-Type'): buf = BytesIO(response.read()) data = gzip.GzipFile(fileobj=buf) elif 'bzip2' in response.info().get('Content-Type'): data = BytesIO(bz2.decompress(response.read())) elif 'zip' in response.info().get('Content-Type'): fzip = zipfile.ZipFile(BytesIO(response.read()), 'r') if len(fzip.namelist())>0: data=BytesIO(fzip.read(fzip.namelist()[0])) return (data, response) # Feeds (NEW) @classmethod def getFeedData(cls, source): source = cls.getFeedURL(source) return cls.getFile(source) if source else None @classmethod def getFeedURL(cls, source): cls.ConfigParser.clear() cls.ConfigParser.read(os.path.join(runPath, "../etc/sources.ini")) return cls.readSetting("Sources", source, cls.sources.get(source, "")) @classmethod def includesFeed(cls, feed): return cls.readSetting("EnabledFeeds", feed, cls.sources.get('include'+feed, False)) # Plugins @classmethod def getPluginLoadSettings(cls): return cls.toPath(cls.readSetting("Plugins", "loadSettings", cls.default['plugin_load'])) @classmethod def getPluginsettings(cls): return cls.toPath(cls.readSetting("Plugins", "pluginSettings", cls.default['plugin_config'])) class ConfigReader(): def __init__(self, file): self.ConfigParser = configparser.ConfigParser() self.ConfigParser.read(file) def read(self, section, item, default): result = default try: if type(default) == bool: result = self.ConfigParser.getboolean(section, item) elif type(default) == int: result = self.ConfigParser.getint(section, item) else: result = self.ConfigParser.get(section, item) except: pass return result RE: Proxy Server - Modules can't connect to URL - nilamo - Nov-16-2016 Holy Jesus that's a lot of classmethods. There's literally no instance methods. That's just a lot of extra code for a fancy looking dict named "Configuration", that has accessors via dot notation. Use the most recent Tagged release. The master branch appears to have been broken in this regard. The fix appears to have been committed, but it just isn't part of the master branch yet. Such is the pitfall of living on the bleeding edge, and why releases exist. RE: Proxy Server - Modules can't connect to URL - robmog - Nov-16-2016 Thanks Nilamo - I haven't seen classmethods anywhere before. I've got more reading to do as the github has got a docs folder with links. For clarity it might help if I paste the the script I was running in the first place "db_mgmt.py" that threw up the Traceback for Configuration.getCVEDict() #!/usr/bin/env python3 # -*- coding: utf-8 -*- # # Manager for the database # # Copyright (c) 2012 Wim Remes # Copyright (c) 2012-2014 Alexandre Dulaunoy - [email protected] # Copyright (c) 2014-2016 Pieter-Jan Moreels - [email protected] # Imports # make sure these modules are available on your system import os import sys runPath = os.path.dirname(os.path.realpath(__file__)) sys.path.append(os.path.join(runPath, "..")) import argparse import datetime from xml.sax import make_parser from xml.sax.handler import ContentHandler from dateutil.parser import parse as parse_datetime from lib.ProgressBar import progressbar from lib.Toolkit import toStringFormattedCPE from lib.Config import Configuration import lib.DatabaseLayer as db # parse command line arguments argparser = argparse.ArgumentParser(description='populate/update the local CVE database') argparser.add_argument('-u', action='store_true', help='update the database') argparser.add_argument('-p', action='store_true', help='populate the database') argparser.add_argument('-a', action='store_true', default=False, help='force populating the CVE database') argparser.add_argument('-f', help='process a local xml file') argparser.add_argument('-v', action='store_true', help='verbose output') args = argparser.parse_args() # init parts of the file names to enable looped file download file_prefix = "nvdcve-2.0-" file_suffix = ".xml.gz" file_mod = "modified" file_rec = "recent" # get the current year. This enables us to download all CVE's up to this year :-) date = datetime.datetime.now() year = date.year + 1 # default config defaultvalue = {} defaultvalue['cwe'] = "Unknown" cveStartYear = Configuration.getCVEStartYear() # define the CVE parser. Thanks to Meredith Patterson (@maradydd) for help on this one. class CVEHandler(ContentHandler): def __init__(self): self.cves = [] self.inCVSSElem = 0 self.inSUMMElem = 0 self.inDTElem = 0 self.inPUBElem = 0 self.inAccessvElem = 0 self.inAccesscElem = 0 self.inAccessaElem = 0 self.inCVSSgenElem = 0 self.inImpactiElem = 0 self.inImpactcElem = 0 self.inImpactaElem = 0 def startElement(self, name, attrs): if name == 'entry': self.cves.append({'id': attrs.get('id'), 'references': [], 'vulnerable_configuration': [], 'vulnerable_configuration_cpe_2_2':[]}) self.ref = attrs.get('id') elif name == 'cpe-lang:fact-ref': self.cves[-1]['vulnerable_configuration'].append(toStringFormattedCPE(attrs.get('name'))) self.cves[-1]['vulnerable_configuration_cpe_2_2'].append(attrs.get('name')) elif name == 'cvss:score': self.inCVSSElem = 1 self.CVSS = "" elif name == 'cvss:access-vector': self.inAccessvElem = 1 self.accessv = "" elif name == 'cvss:access-complexity': self.inAccesscElem = 1 self.accessc = "" elif name == 'cvss:authentication': self.inAccessaElem = 1 self.accessa = "" elif name == 'cvss:confidentiality-impact': self.inImpactcElem = 1 self.impactc = "" elif name == 'cvss:integrity-impact': self.inImpactiElem = 1 self.impacti = "" elif name == 'cvss:availability-impact': self.inImpactaElem = 1 self.impacta = "" elif name == 'cvss:generated-on-datetime': self.inCVSSgenElem = 1 self.cvssgen = "" elif name == 'vuln:summary': self.inSUMMElem = 1 self.SUMM = "" elif name == 'vuln:published-datetime': self.inDTElem = 1 self.DT = "" elif name == 'vuln:last-modified-datetime': self.inPUBElem = 1 self.PUB = "" elif name == 'vuln:reference': self.cves[-1]['references'].append(attrs.get('href')) elif name == 'vuln:cwe': self.cves[-1]['cwe'] = attrs.get('id') def characters(self, ch): if self.inCVSSElem: self.CVSS += ch if self.inSUMMElem: self.SUMM += ch if self.inDTElem: self.DT += ch if self.inPUBElem: self.PUB += ch if self.inAccessvElem: self.accessv += ch if self.inAccesscElem: self.accessc += ch if self.inAccessaElem: self.accessa += ch if self.inCVSSgenElem: self.cvssgen += ch if self.inImpactiElem: self.impacti += ch if self.inImpactcElem: self.impactc += ch if self.inImpactaElem: self.impacta += ch def endElement(self, name): if name == 'cvss:score': self.inCVSSElem = 0 self.cves[-1]['cvss'] = self.CVSS if name == 'cvss:access-vector': self.inAccessvElem = 0 if 'access' not in self.cves[-1]: self.cves[-1]['access'] = {} self.cves[-1]['access']['vector'] = self.accessv if name == 'cvss:access-complexity': self.inAccesscElem = 0 if 'access' not in self.cves[-1]: self.cves[-1]['access'] = {} self.cves[-1]['access']['complexity'] = self.accessc if name == 'cvss:authentication': self.inAccessaElem = 0 if 'access' not in self.cves[-1]: self.cves[-1]['access'] = {} self.cves[-1]['access']['authentication'] = self.accessa if name == 'cvss:confidentiality-impact': self.inImpactcElem = 0 if 'impact' not in self.cves[-1]: self.cves[-1]['impact'] = {} self.cves[-1]['impact']['confidentiality'] = self.impactc if name == 'cvss:integrity-impact': self.inImpactiElem = 0 if 'impact' not in self.cves[-1]: self.cves[-1]['impact'] = {} self.cves[-1]['impact']['integrity'] = self.impacti if name == 'cvss:availability-impact': self.inImpactaElem = 0 if 'impact' not in self.cves[-1]: self.cves[-1]['impact'] = {} self.cves[-1]['impact']['availability'] = self.impacta if name == 'cvss:generated-on-datetime': self.inCVSSgenElem = 0 self.cves[-1]['cvss-time'] = parse_datetime(self.cvssgen, ignoretz=True) if name == 'vuln:summary': self.inSUMMElem = 0 self.cves[-1]['summary'] = self.SUMM if name == 'vuln:published-datetime': self.inDTElem = 0 self.cves[-1]['Published'] = parse_datetime(self.DT, ignoretz=True) if name == 'vuln:last-modified-datetime': self.inPUBElem = 0 self.cves[-1]['Modified'] = parse_datetime(self.PUB, ignoretz=True) if __name__ == '__main__': parser = make_parser() ch = CVEHandler() parser.setContentHandler(ch) # start here if it's an update. if args.u: # get the 'modified' file getfile = file_prefix + file_mod + file_suffix try: (f, r) = Configuration.getFile(Configuration.getFeedURL('cve') + getfile) except: sys.exit("Cannot open url %s. Bad URL or not connected to the internet?"%(Configuration.getFeedURL("cve") + getfile)) i = db.getInfo("cve") last_modified = parse_datetime(r.headers['last-modified'], ignoretz=True) if i is not None: if last_modified == i['last-modified']: print("Not modified") sys.exit(0) db.setColUpdate("cve", last_modified) # get your parser on !! parser = make_parser() ch = CVEHandler() parser.setContentHandler(ch) parser.parse(f) for item in ch.cves: # check if the CVE already exists. x = db.getCVE(item['id']) # if so, update the entry. if x: if 'cvss' not in item: item['cvss'] = None if 'cwe' not in item: item['cwe'] = defaultvalue['cwe'] db.updateCVE(item) else: db.insertCVE(item) # get the 'recent' file getfile = file_prefix + file_rec + file_suffix try: (f, r) = Configuration.getFile(Configuration.getFeedURL('cve') + getfile) except: sys.exit("Cannot open url %s. Bad URL or not connected to the internet?"%(Configuration.getFeedURL("cve") + getfile)) parser = make_parser() ch = CVEHandler() parser.setContentHandler(ch) parser.parse(f) for item in progressbar(ch.cves): # check if the CVE already exists. x = db.getCVE(item['id']) # if so, update the entry. if x: if args.v: print("item found : " + item['id']) if 'cvss' not in item: item['cvss'] = None else: item['cvss'] = float(item['cvss']) if 'cwe' not in item: item['cwe'] = defaultvalue['cwe'] db.updateCVE(item) # if not, create it. else: db.insertCVE(item) elif args.p: # populate is pretty straight-forward, just grab all the files from NVD # and dump them into a DB. c = db.getSize('cves') if args.v: print(str(c)) if c > 0 and args.a is False: print("database already populated") else: print("Database population started") for x in range(cveStartYear, year): parser = make_parser() ch = CVEHandler() parser.setContentHandler(ch) getfile = file_prefix + str(x) + file_suffix try: (f, r) = Configuration.getFile(Configuration.getFeedURL('cve') + getfile) except: sys.exit("Cannot open url %s. Bad URL or not connected to the internet?"%(Configuration.getCVEDict() + getfile)) parser.parse(f) if args.v: for item in ch.cves: print(item['id']) for item in ch.cves: if 'cvss' in item: item['cvss'] = float(item['cvss']) # check if year is not cve-free if len(ch.cves) != 0: print("Importing CVEs for year " + str(x)) ret = db.insertCVE(ch.cves) else: print ("Year " + str(x) + " has no CVE's.") RE: Proxy Server - Modules can't connect to URL - nilamo - Nov-16-2016 ...is it good now? lol RE: Proxy Server - Modules can't connect to URL - robmog - Nov-17-2016 No this is way out of my league! I'm going to have read up on python @classmethods and then rtfm again and maybe then I can ask one of the team members listed on the github project page. I will report back soon. Thanks again I appreciate your help. RE: Proxy Server - Modules can't connect to URL - robmog - Nov-17-2016 Hello Again - just to update - I had a chat with the team member responsible for that bit of code and they said it's a bug and they'll sort it asap. |