pydl.ThreadProc Class Reference

Inheritance diagram for pydl.ThreadProc:

Collaboration diagram for pydl.ThreadProc:

Public Member Functions
def	__init__ (self, urlq, tq, dl_site_info, report)
def	run (self)

Public Attributes
	urlq
	tq
	dl_site_info
	report

Detailed Description

Definition at line 191 of file pydl.py.

Constructor & Destructor Documentation

def pydl.ThreadProc.__init__	(		self,
			urlq,
			tq,
			dl_site_info,
			report
	)

Definition at line 193 of file pydl.py.

    def __init__(self, urlq, tq, dl_site_info, report):
        Thread.__init__(self)
        self.urlq = urlq
        self.tq = tq
        self.dl_site_info = dl_site_info
        self.report = report

Member Function Documentation

def pydl.ThreadProc.run

(

self

)

Definition at line 200 of file pydl.py.

References pydl.ThreadProc.urlq.

    def run(self):
        try:
            url = self.urlq.get()
            while True:
                if self.urlq == None:
                    break;

                # setup download_path link portion
                download_path = None
                id_pos = None

                # grab the base download url based on the patterns provided
                for key, value in dl_site_info.items():
                    if ( key in url ):
                        download_path = value["download_path"]
                        # grab the file id position from the template
                        id_pos = value["id_pos"]

                # TODO: needs error handling -exit bad
                if( download_path == None ):
                    print("ERROR: base url of website not configured")
                    # TODO: should probably just continue here
                    sys.exit(1)

                # create object and do work
                try:
                    o = ScrapeFile(url, download_path, id_pos, report)
                    report.progressX[o.ofilename] = {"bytes_so_far": 0, "total_size": 0}
                    if DEBUG:
                        o.test_thread()
                    else:
                        o.download()
                    self.tq.get()
                    self.urlq.task_done()
                    break;
                except urllib.error.HTTPError as err:
                    print("ERROR: %s | HTTPError: %s" % (url, err))
                    continue

        except MyError as err:
            print("\n" + "** signal caught: %s **\n" % err)
            exit(1)

        #self.tq.task_done()

Member Data Documentation

pydl.ThreadProc.dl_site_info

Definition at line 197 of file pydl.py.

pydl.ThreadProc.report

Definition at line 198 of file pydl.py.

pydl.ThreadProc.tq

Definition at line 196 of file pydl.py.

pydl.ThreadProc.urlq

Definition at line 195 of file pydl.py.

Referenced by pydl.ThreadProc.run().

---

The documentation for this class was generated from the following file:

• pydl.py