Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add apk url and version code for latest release #6

Merged
merged 4 commits into from
Jul 4, 2018
Merged

Add apk url and version code for latest release #6

Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b131ed8
Add apk url for latest GitHub release as "github_apk"
TobiGr Jul 4, 2018
dd555dc
Add F-Droid apk and version code
TobiGr Jul 4, 2018
49726f1
make version_code a number
TobiGr Jul 4, 2018
418d486
restructure JSON output
TobiGr Jul 4, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 39 additions & 7 deletions api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,8 @@ def fetch_data_and_assemble_response(self):

repo_url = "https://api.github.com/repos/TeamNewPipe/NewPipe"

release_github_url = "https://github.com/TeamNewPipe/NewPipe/releases/"

contributors_url = "https://github.com/TeamNewPipe/NewPipe"

translations_url = "https://hosted.weblate.org/api/components/" \
Expand All @@ -121,6 +123,7 @@ def fetch(request: tornado.httpclient.HTTPRequest):
responses = yield tornado.gen.multi((
fetch(make_request(repo_url)),
fetch(make_request(stable_url)),
fetch(make_request(release_github_url)),
fetch(make_request(contributors_url)),
fetch(make_request(translations_url)),
))
Expand All @@ -130,21 +133,44 @@ def fetch(request: tornado.httpclient.HTTPRequest):
self.__class__._last_failed_request = datetime.now()
return False

repo_data, stable_data, \
repo_data, stable_data, release_github_data, \
contributors_data, translations_data = [x.body for x in responses]

def assemble_release_data(data: str):
if isinstance(data, bytes):
data = data.decode()

versions = re.findall("commit=(.*)", data)
def assemble_release_data(version_data: str):
if isinstance(version_data, bytes):
version_data = version_data.decode()

versions = re.findall("commit=(.*)", version_data)
version_codes = re.findall("Build:(.*)", version_data)
version_code = version_codes[-1].split(",")[-1]
return {
"version": versions[-1],
"version_code": int(version_code),
"apk": "https://f-droid.org/repo/org.schabi.newpipe_" + version_code + ".apk",
}

repo = json.loads(repo_data)

# scrap latest GitHub version and apk from website
elem = html.fromstring(release_github_data)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TheAssassin Do you have any idea how I can reduce the redundancy here? I didn't find a good way to do so.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What kind of redundancy are you talking about specifically?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have the same structure for getting the GitHub version, apk and the contributors count:
Get HTML, use cssselect to filter the HTML, check if the we get the a valid number of results, get the first result and modify it.

Unfortunately I couldn't move this to a new method because I failed to execute the steps in the try blocks in a good way. My initial idea was to return tags[0]:

        def scrape_data_from_html(data, selector: str):
            elem = html.fromstring(data)
            tags = elem.cssselect(selector)
            if len(tags) < 1:
                return -1
            else:
                try:
                    return tags[0]
                except:
                    return -1

But than I realised that I'd needed another try and catch to handle the possible exceptions which can be thrown by tags[0].get('href'), tags[0].text and int(tags[0].text):

contributors_result = scrape_data_from_html(contributors_data, ".numbers-summary a[href$=contributors] .num")
try:
    contributors_count = int(contributors_result.text)
except:
    contributors_count = -1

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am okay with this as it is.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good,

tags = elem.cssselect('.release-body li.d-block a[href$=".apk"]')
if len(tags) == 0:
release_github_apk = -1
else:
try:
release_github_apk = "https://github.com" + tags[0].get('href')
except:
release_github_apk = -1
tags = elem.cssselect('.release-header h1.release-title a')
if len(tags) == 0:
release_github_version = -1
else:
try:
release_github_version = tags[0].text
except:
release_github_version = -1

# scrap contributors from website
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's "scrape" ("extract"), not "scrap" ("dispose", "throw away").

elem = html.fromstring(contributors_data)
tags = elem.cssselect(".numbers-summary a[href$=contributors] .num")
if len(tags) != 1:
Expand All @@ -166,7 +192,13 @@ def assemble_release_data(data: str):
"translations": int(translations["count"]),
},
"flavors": {
"stable": assemble_release_data(stable_data),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to split this up into "github": {"stable": {...}, ...} and "f-droid": {"stable": {...}, ...} instead? GitHub names releases differently, i.e. there's releases and pre-releases.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good.

"stable": {
"f-droid": assemble_release_data(stable_data),
"github": {
"version": release_github_version,
"apk": release_github_apk,
}
}
}
}

Expand Down